LeaderRebalancerTest.AddTserver: /home/jenkins-slave/workspace/build_and_test@3/src/kudu/master/auto_leader_rebalancer-test.cc:427: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 1
I20240426 13:22:44.572235 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
W20240426 13:22:44.591799 25069 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:44.550753   956 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.633641  1010 raft_consensus.cc:1050] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:44.635990 25618 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:44.643568 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 171976us
W20240426 13:22:44.653997 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.725553  1072 ts_tablet_manager.cc:1888] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.16
I20240426 13:22:44.726132  1072 log.cc:1196] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/0d8fda82f95348219809212d99d6704e
I20240426 13:22:44.754010 24847 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:44.764339 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.770047 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44116.5us
I20240426 13:22:44.774834 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 0d8fda82f95348219809212d99d6704e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:44.783321 25023 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.818358  1010 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:44.828127  1073 ts_tablet_manager.cc:1888] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 22.17
I20240426 13:22:44.828678  1073 log.cc:1196] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/f7408112bace482693c999f678aa9d73
I20240426 13:22:44.830314 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet f7408112bace482693c999f678aa9d73 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
W20240426 13:22:44.830626 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.833690 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.843075 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.848208 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.848862 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.849603 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.850190 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.850806 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.851487 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.852224 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.852880 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.853556 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.859182 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60218.4us
W20240426 13:22:44.880656 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.888876 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.892140 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.894915 25105 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20240426 13:22:44.895104 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.899225 25618 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20240426 13:22:44.898768 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.901017  1010 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
W20240426 13:22:44.905922 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.909533 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.910607  1104 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
W20240426 13:22:44.919026 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.924908 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.927137 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.940552 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.950642 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.959137 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.991379  1104 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.993305 25622 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.033598 25101 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.068442 25105 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.088595 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:45.092676 25622 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20240426 13:22:45.099251 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.100855  1097 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
W20240426 13:22:45.111652 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.164762  1094 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.165541 25622 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:45.181263 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.188351  1080 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.188946  1080 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.190910  1080 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.193002 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.194702 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.196800 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.198643  1080 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.211783 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9108945c08ba4b2287c1e238b402722f with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:45.212304  1083 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.212860  1083 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.215037  1083 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.217613 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.221398 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.223385 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.224578  1085 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.239740  1085 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.241562  1085 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.243409  1085 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.249439 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.251071 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.265107 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.266422  1083 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.287696 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:45.284451 24846 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.294436 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.296803 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.298807 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.304309 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.305140 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.305745 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.313620 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.315790  1100 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.318962 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.320134 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.316445  1100 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.322733 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.323273  1100 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.323606 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.324975 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.325974 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.329408 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9108945c08ba4b2287c1e238b402722f with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:45.327564 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.338201  1118 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:45.378266 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.379112  1085 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.379773  1085 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.380990 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.381893  1085 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.382764 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.383980  1119 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:45.381745 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.385051  1119 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.385144 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.386487 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.388260 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.389183 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.389864 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.391989  1083 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:45.389977 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.399096  1119 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.403944  1119 ts_tablet_manager.cc:1875] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:45.405019 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.407445 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.408450  1083 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.409404 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.409019  1083 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.414006  1085 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.415117  1085 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.419076 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.420521 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.422165 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.422422  1085 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.422581 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.423080  1106 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:45.423939 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.424340  1083 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.424939 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.425757 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.427026 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.428485 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.429797  1106 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:45.430470  1106 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.430990  1106 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:45.432276 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.433069  1106 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.434535 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.437714  1085 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.440475  1085 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.434535 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.443850 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.444875  1085 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.445744 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.445755 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.448004 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.449033  1106 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:45.445452  1085 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.450843 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.451736 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.467657  1106 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.468294  1106 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.470352  1106 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.476539 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.479945 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.482333  1119 ts_tablet_manager.cc:1888] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.16
I20240426 13:22:45.482841  1119 log.cc:1196] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/9108945c08ba4b2287c1e238b402722f
I20240426 13:22:45.483029  1083 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.483556  1083 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.484277 24920 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.485433  1083 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.485944  1083 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:45.487643  1118 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.488149  1118 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.490024  1118 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.490128 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.491389 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.492630 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.487098 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 9108945c08ba4b2287c1e238b402722f (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:45.495515 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.495527 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.496901  1085 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:45.497200 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.498289  1085 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:45.459620 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.502056 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.514619 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.515748  1118 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.516309  1118 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.517678 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.518210  1118 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.518698 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.519407 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.523546 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.531911  1118 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.532496  1118 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.539445 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.536440 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.547163  1118 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.546465 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.582475 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.583776  1118 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:45.586220 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.587163 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.587814 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.650810 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.599182 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:45.652664 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 53558.3us
I20240426 13:22:45.651813 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.669633 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.672371 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.675094  1118 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:22:45.678503 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.679111 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.686054 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.686882 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.687768 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.691609 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.692402 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.692926 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.695467 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.719192 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.722203 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.729033 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.729899 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.735805 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.742772 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.743548 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.744129 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.752629 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.756510 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.757210 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.761301 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.762115 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.763177 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.799274 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.800017 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.801122 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.803732 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.806157 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.825973 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.834414 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.835345 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.836014 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.847889 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.848788 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.849542 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.852772 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.853610 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.854187 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.855971  1085 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.856544  1085 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.857007 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.857877 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.858422 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.858503  1085 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.860991 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.862632 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.863318  1085 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:45.863451 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.864266 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.863986  1085 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.867025 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.867461 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.867847 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.868592 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.866278  1085 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.871915 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.874307 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.877468 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.881847 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.889531 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.890764 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.894011 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.894701 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.900086 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.909188 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.906903 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.910369  1085 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:45.912587 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.913401 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.914062 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.895584 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.915953 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.916950  1118 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:45.925469 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.926429 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.926999 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.930102 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.937103 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.937781 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.942082 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.946494 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.947643 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.950217 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.950990 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.951675 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.966591 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.967413 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.968502 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.977528 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.978313 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.978906 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.985679 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.988626 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.989727 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.992524 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.993221 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.993755 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.996222 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.996845 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.007515 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.013038 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.015576 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.017992 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.021683 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.026161 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:46.031155 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.034119 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.034842 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.044942 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.048658 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.049250 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:46.049822 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.390157  1125 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.390718  1125 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.392827  1125 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:46.396026 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:46.396272 25069 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:46.397589 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.400100  1122 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:46.651064  1124 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.651690  1124 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.653949  1124 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.658227 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.658947 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:46.660270 25589 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:46.663486  1121 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:46.663893  1121 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:46.664503 25588 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.664250  1121 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:46.681870  1010 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.682533  1010 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.688669  1121 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.692646  1010 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:46.703469 25588 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.704640 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:46.705597 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:46.706224 25105 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:46.712138  1121 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.714277 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:46.723121 25589 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.724983  1122 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.725534  1122 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.724529  1121 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:46.728843  1122 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.729410  1121 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:46.729887 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.731367  1121 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.760318  1130 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.760903  1130 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.763398  1130 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.770236  1124 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.770845  1124 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.773355  1124 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.775213 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.777459 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.781038 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
W20240426 13:22:46.779714 25588 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.786375  1084 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.786896  1084 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.790020 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.791589  1084 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.795672 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.797032  1084 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:46.797439  1084 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.797849  1084 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:46.798106 25588 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.798878  1130 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.799490  1130 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.801393  1130 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.802259  1122 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.802853  1122 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.805846 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.807442  1122 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.807894 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:46.810415 25069 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.810879 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:46.811908  1026 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:46.809312  1124 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:46.813400 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:46.813513 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:46.814033 25098 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:46.834709  1084 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.839669  1010 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.840287  1010 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.842340  1010 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.851601 24849 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 21 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:46.854681  1084 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.859118 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.859252  1010 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.859853  1010 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.861845  1010 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:46.866957 24995 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.867458 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.868721  1130 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:46.869158  1130 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.869525  1130 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:46.872233 25588 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.877481  1010 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:46.877142  1130 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.878631  1122 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.879388  1122 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.884229 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.884867 25023 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:46.887671 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.888589  1026 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:46.888978  1026 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.889303  1026 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:46.890239 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.896531 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.897258 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:46.880208  1130 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.900482 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:46.901088 25105 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:46.904275 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:46.905965 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:46.907619 25022 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:46.911576 25069 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.915279 25022 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:46.916435  1122 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.923249  1026 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.926561 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:46.927284 25022 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:46.935078 25098 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:46.949766 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.950626  1010 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:46.951110  1010 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.951550  1010 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:46.912091 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:46.954414  1122 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:46.955446 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43894.9us
I20240426 13:22:46.959931  1010 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:46.964053 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.984791 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:46.985639  1104 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20240426 13:22:46.987452 24995 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.981695 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.988685  1121 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:46.989190  1121 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:46.990093  1121 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.981096  1084 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.991667  1084 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.994865 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:47.001405 25588 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.001953 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.002784  1124 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:47.003731  1084 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.981928 25105 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
I20240426 13:22:46.965620  1026 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.006170 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.006951  1130 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:47.007675  1130 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.008745  1130 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.014861 25069 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.016923  1010 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.018649 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.037293 25105 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.025817  1122 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.038620  1122 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.039548   994 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.040210   994 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.042063   994 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.042383  1122 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.043244 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.043989 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 25.
I20240426 13:22:47.046207 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.047019   994 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:47.047461   994 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.047041 24846 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 25 to 26, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 26 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.047870   994 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:47.052207 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.063496   994 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.069772 25069 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.072741 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.075223 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.075946 25619 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
W20240426 13:22:47.077858 24995 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.093431 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.081908 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.108371 25028 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:47.032207 25022 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:47.112074 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.112828  1130 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:47.089638   994 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.094635  1125 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.114738  1026 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:47.114830  1125 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.115576  1026 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.116618  1026 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.117149  1125 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.079041 25069 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.104187  1010 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.123250  1010 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.105408 24849 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 24 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.133239  1010 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.104749 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.148249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43542.8us
W20240426 13:22:47.135784 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.138065 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.106793 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:47.154770  1122 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:47.155236  1122 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.155629  1122 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.159262 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:47.161624 25069 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.162616 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.164268  1122 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.164460 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.167006 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.155488 24995 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.167382  1130 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:47.168841  1130 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.169724  1130 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.177000  1122 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.188897 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.192643 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
W20240426 13:22:47.197515 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.200984 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.201920  1122 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:47.202646  1122 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.203440  1122 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.204584 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.205246 25619 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.215480  1084 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.216109  1084 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.218113  1084 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.224606  1121 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.223766 24849 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 24 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 25 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.225153  1121 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.225786  1084 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.226850  1084 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.229255  1084 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.231071  1121 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.189028 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.234270 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45268.3us
W20240426 13:22:47.242152 25588 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.249349  1121 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.249640 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.249930  1121 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.250597 25105 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 26.
I20240426 13:22:47.253131 25589 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.254474  1084 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:22:47.254956  1084 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.255367  1084 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:47.256071   994 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.257012  1125 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.257485  1125 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.259409  1125 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.256599   994 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.262022   994 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.264837  1121 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.270851 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.284587  1124 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.304980  1124 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.304572 25619 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
W20240426 13:22:47.287608 25069 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.291636  1130 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.309900  1130 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.311709 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.311977  1130 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.312800 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.314692 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.320334 25588 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.321197 25105 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20240426 13:22:47.313226 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.323204  1122 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.323871  1122 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.323277 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.325457 24995 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.325824  1122 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.325945 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:47.299993 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101265us
I20240426 13:22:47.306975  1124 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.286113 24846 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:22:47.326566 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9960 after lost signal to thread 23799
W20240426 13:22:47.330992 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d99e0 after lost signal to thread 24888
I20240426 13:22:47.331496  1130 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:22:47.332588 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9f60 after lost signal to thread 25055
W20240426 13:22:47.330916 25069 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.333889 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9540 after lost signal to thread 25129
I20240426 13:22:47.333976  1104 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.336570  1104 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.338907  1104 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.340371  1124 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:47.341040 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08001a7900 after lost signal to thread 25650
I20240426 13:22:47.340911  1124 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.344962  1124 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.348228 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.349086  1084 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.353034  1124 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:47.361093 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.380769 25617 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:47.361234 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.363015 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.365065  1084 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.366183 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.385628 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.383100  1130 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.386265 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:47.386590  1130 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.387310 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.388535  1130 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.390239 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.390463  1010 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.381803 25615 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:47.390899  1010 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.371541  1122 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:47.372709 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.392406 25030 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:47.373724 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.375360 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.394001 25022 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:47.384491 24846 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 25 to 26, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 26 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.382603 25028 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 27.
I20240426 13:22:47.394443  1010 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.394966  1010 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20240426 13:22:47.395243  1010 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.395527  1010 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Advancing to term 28
W20240426 13:22:47.396078 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.401098 25588 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.401479 25105 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:47.415341 25105 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 27.
I20240426 13:22:47.416085 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.417044 25026 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:47.421221 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.421964 25617 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:47.422932 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.435133 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.421914 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.436313 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.439299  1125 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20240426 13:22:47.441046 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64000.1us
W20240426 13:22:47.441659 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:47.442333 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.442613 25589 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:47.442947 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.440447  1010 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.444272  1125 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.444701  1125 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:47.445171  1124 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:47.445708  1124 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:47.446606  1124 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.439658  1026 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:47.448455  1026 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.448721  1142 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20240426 13:22:47.449146  1142 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.449235  1026 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.450400 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.450851 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.451586 24995 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.449826  1142 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:47.453011 25028 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 28: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.453871 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.450899 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.456599   990 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20240426 13:22:47.457363 25588 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.458099 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.458788 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64253.2us
I20240426 13:22:47.459978  1010 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.461052  1010 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:47.462237 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.458127  1104 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.464170 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.464969 25589 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.461607  1010 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.467103   990 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:47.463862  1121 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:47.469702  1121 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.470060  1121 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:47.472903 25589 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.467990  1010 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:47.473928  1125 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.479863 25098 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:47.472368  1104 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.482353  1125 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.485216 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.489640  1104 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.493551  1142 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.495422 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57991.5us
I20240426 13:22:47.483448  1121 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.506052 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.509219 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.510109 25098 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Advancing to term 28
W20240426 13:22:47.514837 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51506.2us
W20240426 13:22:47.516213 24995 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.517132 24995 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.517714 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.520272 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.520889 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:47.523105 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:47.520916 25617 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
W20240426 13:22:47.523869 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.524452 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.524966 24995 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.526791 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.508599  1142 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.541373 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.538674  1010 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.547391  1145 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20240426 13:22:47.535923 23796 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:47.561497 25069 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.509094   990 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:47.562420  1010 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.563159 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 98275.9us
I20240426 13:22:47.531582  1084 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20240426 13:22:47.568228 25069 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.570785 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.571259 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.571728 25069 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.572862 25069 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.564179   990 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.576798   990 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.580987 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 28.
I20240426 13:22:47.574860 24846 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 18 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.584247  1151 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:47.584697  1151 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.585036  1151 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.547943  1145 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.586361  1145 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:47.583657   990 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.594108 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:47.597352 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 69021.9us
I20240426 13:22:47.601694 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.603654  1130 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:47.640970  1156 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:47.641561 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.644776  1145 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.648581  1145 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.572635  1142 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.594146  1151 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.572654  1084 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:47.660303 25588 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.666203 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 139495us
W20240426 13:22:47.666900 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.667449 24995 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.674597 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.654664   990 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.682228 25617 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:47.650583  1157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election won for term 28
I20240426 13:22:47.651322  1130 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:47.684877 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.685647 25619 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.666777  1084 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:47.657824  1121 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.659621  1151 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.703684 25619 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
W20240426 13:22:47.720978 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 126945us
I20240426 13:22:47.652372  1142 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.725584 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 213269us
I20240426 13:22:47.720172 24846 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 26 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 27 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:47.734434 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.760866 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.765120 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.766008 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 23: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.765864  1084 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.768916 25027 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20240426 13:22:47.771344  1142 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.773447 25615 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 28: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:47.773466 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.774400 25030 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.778461 25023 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20240426 13:22:47.778992 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.779518 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:47.781131  1084 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.791013 25098 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20240426 13:22:47.800457 25023 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20240426 13:22:47.798053 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52148.2us
I20240426 13:22:47.807170 25022 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:47.809711 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:47.810777 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.811390 25023 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 28: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.816637 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.830961 25589 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.831838 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.832026 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 235182us
W20240426 13:22:47.833490 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.835281  1125 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:47.835901  1125 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.836863  1125 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.824980  1145 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20240426 13:22:47.815339  1147 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20240426 13:22:47.816893  1157 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.842193  1122 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:47.847139 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.847707 25615 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:47.823702  1084 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:47.851578  1143 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:47.852123  1143 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:47.836184  1124 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20240426 13:22:47.853183  1143 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.859438 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.859728 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.860201 25619 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
I20240426 13:22:47.870064 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.870818 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.871417 25069 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.872481 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.872825  1010 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:47.881889 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.882723  1010 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.885298 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.886154  1010 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.879240  1121 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20240426 13:22:47.843973  1157 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.892760  1166 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:22:47.886824  1161 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20240426 13:22:47.909265 24995 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.909165  1166 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.911343 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.922973 25615 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:47.924584  1161 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:47.920943  1166 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
W20240426 13:22:47.935683 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.947427 25069 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.948298 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.951310 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.955298 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:47.958271 24995 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.963181 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.963464 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.952844 24846 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.964310 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:47.977113 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92826.4us
I20240426 13:22:48.010300  1146 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
W20240426 13:22:48.050143 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.054149  1146 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:48.055325  1146 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:48.072680  1171 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20240426 13:22:48.077335  1170 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20240426 13:22:48.086481 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.111093 24849 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 19 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:48.129063 25098 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20240426 13:22:48.147572 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104881us
I20240426 13:22:48.152639  1166 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:48.162236  1166 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:48.148905  1154 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20240426 13:22:48.170400 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:48.171039 25617 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:48.176697 25619 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20240426 13:22:48.176873 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.182552  1165 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20240426 13:22:48.183697  1173 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20240426 13:22:48.194216 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:48.197628 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:48.201418 25101 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:48.220739  1158 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:48.204357  1104 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:48.225822  1104 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:48.226758  1104 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:48.283342 24854 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 27 to 28, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 28 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.318696 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:48.319275 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:48.422670 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80343.9us
W20240426 13:22:48.426178 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66793.8us
I20240426 13:22:48.426463 25022 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20240426 13:22:48.432453 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:48.440608 25022 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:22:48.442229  1026 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:22:48.449182 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.463960  1176 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20240426 13:22:48.480574 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.483254 25104 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20240426 13:22:48.485672 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.489725  1158 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:48.494452 25619 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20240426 13:22:48.505865 25619 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:48.507797  1163 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20240426 13:22:48.511231  1150 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:48.526867 24846 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 19 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.559273 23796 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:48.563249 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:48.615526 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.640055 24846 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 24 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 25 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:48.689522 24849 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 26 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 27 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.771582 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.841828 25617 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20240426 13:22:48.843741  1158 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240426 13:22:49.422816 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.428328 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.513195 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.554678 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.559778 23796 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:49.568109 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.575170 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.605594  1176 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:49.606118  1176 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.607985  1176 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:49.610713 25588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111) [suppressed 397 similar messages]
W20240426 13:22:49.629650 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:49.645274 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.648419 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:49.649111 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:49.652993 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.655066  1176 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:49.655472  1176 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:49.655779  1176 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:49.662966 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.663619  1176 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.679800  1176 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:49.683408 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:49.684134 25022 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:49.690914 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
W20240426 13:22:49.692325 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.696507 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.697721  1176 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:49.698493  1176 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:49.699388  1176 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.731081 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.741080 24846 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 22 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 24 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:49.825793 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.839496 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.847167 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.863714  1176 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:49.864326  1176 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.866289  1176 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:49.869601 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:49.870249 25028 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 28.
W20240426 13:22:49.870450 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.873701 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.879549  1176 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20240426 13:22:49.880000  1176 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:49.880378  1176 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Advancing to term 29
W20240426 13:22:49.879555 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.887359 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.904927 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.907675 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.920349 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.910969  1176 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.911649 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.929868 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.925323  1176 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:49.933714 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.935551 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:49.936174 25022 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Advancing to term 29
W20240426 13:22:49.937963 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:49.942744 25022 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 29.
I20240426 13:22:49.944955 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.946118  1176 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader election won for term 29
I20240426 13:22:49.947417  1176 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:49.948817  1176 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.970552 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.985607 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.025058 24846 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 27 to 29, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 29 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:50.086858 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.095738 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.125599 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.136864 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.146849 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.170759 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.229094 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:50.272634 25022 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20240426 13:22:50.274361 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.275416  1208 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240426 13:22:50.305732 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.331508 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.337718 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:50.404390 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.441936 25027 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20240426 13:22:50.443778  1208 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20240426 13:22:50.468705 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:50.503435 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.560295 23796 thread.cc:536] Waited for 4000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:50.575412 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.616986 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.684512 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.828544 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.894778 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.006484 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66299.8us
W20240426 13:22:51.048838 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 116625us
W20240426 13:22:51.054015 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.288707 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 194448us
W20240426 13:22:51.318878 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.331403 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.333138 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.504374 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 165815us
W20240426 13:22:51.525668 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.555200 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 190156us
W20240426 13:22:51.560860 23796 thread.cc:536] Waited for 5000ms trying to join with diag-logger (tid 24977)
I20240426 13:22:51.763971 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033334
I20240426 13:22:51.929883 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.165s	user 0.040s	sys 0.011s Metrics: {"bytes_written":8730,"delete_count":0,"lbm_write_time_us":13340,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":42763,"threads_started":1,"update_count":341,"wal-append.queue_time_us":42398}
I20240426 13:22:51.945315 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.440299
W20240426 13:22:51.939864 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.112576 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.125286 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.130538 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d5a0 after lost signal to thread 23799
W20240426 13:22:52.131749 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d4c0 after lost signal to thread 24888
W20240426 13:22:52.133199 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080035ac00 after lost signal to thread 25055
W20240426 13:22:52.134429 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d100 after lost signal to thread 25129
W20240426 13:22:52.135696 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08003dbd80 after lost signal to thread 25650
W20240426 13:22:52.154394 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.171723 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.267086 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.341208 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:52.340409 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.394s	user 0.126s	sys 0.037s Metrics: {"cfile_cache_miss":10,"cfile_cache_miss_bytes":29850,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":32379,"dirs.run_cpu_time_us":617,"dirs.run_wall_time_us":4899,"lbm_read_time_us":3574,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":25,"lbm_write_time_us":2952,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":11,"reinsert_count":0,"thread_start_us":47390,"threads_started":2,"update_count":369}
I20240426 13:22:52.343142 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20240426 13:22:52.344944 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.355780 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:52.391683 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
W20240426 13:22:52.454952 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.513686 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.534705 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.555557 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.561619 23796 thread.cc:536] Waited for 6000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:52.590241 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.593245 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.624650 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.610005 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.647534 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.648182 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.656507 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.667943 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.705339 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.746249 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.877545 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.913552 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:52.915097 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.926405 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.927022 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.950424 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.994331 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:53.014695 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.016145 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.060734 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.126049 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.128130 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.179775 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.215186 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.277762 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.354686 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.554603 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.563705 23796 thread.cc:536] Waited for 7000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:53.590504 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.605877 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.677711 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.701898 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.798374 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:54.124171 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:54.128654 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47751.4us
W20240426 13:22:54.381165 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.417551 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.507091 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.551371 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.566098 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.612005 25069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111) [suppressed 443 similar messages]
W20240426 13:22:54.777138 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.787890 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:54.816329 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
W20240426 13:22:54.824988 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.846951 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.862830 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.958529 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.003166 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.042971 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.049167 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.058523 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
W20240426 13:22:55.061870 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.082975 25589 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.085101 25070 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.087411 24901 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240426 13:22:55.087817 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240426 13:22:55.095746 25589 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.097926 25589 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.103075 25070 consensus_peers.cc:485] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.121214 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.124385 25589 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.135139 25070 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.140012 25070 consensus_peers.cc:485] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.152022 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.162460 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240426 13:22:55.168269 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.168433 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.169554 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:55.170177 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.171006 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.176803 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.178002 25589 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.177758 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.179658 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.181607 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.182196 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.182682 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.184803 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.185369 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.185914 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.189677 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.190430 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.193840 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.193862 25589 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.195045 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.196784 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.197578 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.198145 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.200352 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.200959 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.201894 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.204681 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.205557 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Raft consensus shutting down.
W20240426 13:22:55.206100 25589 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.206629 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.208681 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.209336 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.210050 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.212396 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.213172 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.214251 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.216740 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.217547 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.218178 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.239518 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.241392 25070 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.241392 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:55.253221 25070 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.253367 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.253912 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.255276 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.257723 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.258371 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.259305 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.267235 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.267294 25070 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.270280 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.271188 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.271764 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.273480 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.274966 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.275804 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.277973 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.279906 25070 consensus_peers.cc:485] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.281563 25070 consensus_peers.cc:485] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.282852 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.283769 25070 consensus_peers.cc:485] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.283735 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.284788 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.287743 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.289109 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.289727 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.290436 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.292125 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.292728 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.293507 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.295959 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.296618 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.297220 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.299552 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.300261 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.300900 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.303443 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.304195 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.305366 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.307932 25589 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.308105 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.308912 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.309430 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.312089 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.312824 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.313452 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.316272 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.317031 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.317991 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.320739 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.321452 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.322613 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.325145 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.325876 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.326611 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.329183 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.329892 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.331071 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.333529 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.334218 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.335268 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.337698 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.338387 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.339850 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.342514 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.343215 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.343860 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.345399 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.346393 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.347661 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.348749 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.351295 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.352000 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.352612 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.355440 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.356174 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.357280 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.359787 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.360587 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.361400 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.364111 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.364898 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.365617 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.368221 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.368897 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.369480 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.372561 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.373253 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.374493 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.377146 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.377792 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.378710 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.380234 25589 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.381242 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.382053 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.383265 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.385815 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.386488 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.387130 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.389748 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.390415 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.391458 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.394034 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.394690 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.395718 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.397523 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.398419 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.399207 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.399749 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.402374 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.403018 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.405241 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.407961 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.409482 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.412611 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.413786 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.416317 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.416954 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.418037 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.418037 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:55.419570 25070 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.420727 25070 consensus_peers.cc:485] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.432413 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.445542 25070 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.458009 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:55.474308 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.490551 25070 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.543438 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.552938 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:55.568991 25070 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.606221 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.664830 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:55.750147 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:56.208699 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.221256 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.221609 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.228353 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.228685 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.237027 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5b1db66b7c1043e1bd43d7254a4dec62, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.237392 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5b1db66b7c1043e1bd43d7254a4dec62, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.247519 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.247857 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.254961  1221 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.255630  1221 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.257695  1221 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.260210 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:56.264151 25589 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.264626 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.265616  1221 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20240426 13:22:56.274752  1221 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.275303  1221 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.277487  1221 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.281030 25588 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.282740 25589 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.283138 25589 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.283800  1221 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20240426 13:22:56.315795 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.340577  1221 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.341293  1221 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.343662  1221 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.348667 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:56.349577 25589 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.351888 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.353331  1221 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:56.367038  1247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.367566  1247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.369514  1247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.372997 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.373665 25622 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
W20240426 13:22:56.374845 25070 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.376094 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.377229  1247 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:56.377631  1247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.370874  1221 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.378037  1247 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:56.380599  1221 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.382854  1221 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.389921 25588 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.393209 25589 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.393682 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.395493  1221 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:56.400321  1216 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.400835  1216 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.402886  1216 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.403085  1247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.405189  1247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.412384 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:56.413059 25621 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:56.413244 25070 leader_election.cc:334] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.413758 25070 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.417088 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:22:56.419667 25621 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
W20240426 13:22:56.420517 25069 leader_election.cc:334] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.420893 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.421460  1247 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:56.421993 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.422746  1247 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:56.423215  1247 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:56.424203  1247 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.432740 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:56.438757 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.445848  1247 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.446558  1247 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.450695 24846 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:56.451642  1216 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.452257  1216 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.454207  1216 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.454638  1247 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.456362  1247 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.456852  1221 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.456892  1247 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.457464  1221 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.459818  1247 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.462865 25070 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.466444 25070 leader_election.cc:334] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.477201  1221 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.477861 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.480079 25589 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.481488 25070 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.483209 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.483706 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.480569 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.488076  1221 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:56.491050 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.493240 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.493969  1216 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:56.494511  1216 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.495568 25069 leader_election.cc:334] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.495807  1221 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.496016 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.497730  1247 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.498682  1221 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.500157  1243 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.501029  1221 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.500845  1243 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.503418  1243 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.505976 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:56.506718 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:56.508999 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:56.509665 25098 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:56.513554 25589 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:56.514458  1243 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:56.514935  1243 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.515348  1243 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:56.524107 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:56.525149  1221 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:56.525574  1221 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.525974  1221 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:56.538985  1247 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.534440  1221 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.539541  1247 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.541368  1221 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.535733 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.541788  1247 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.542502 25589 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.523211  1243 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.547808  1221 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.548431  1221 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.551504 25070 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.552326 25069 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.552170 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:56.552778 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.552947 25098 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:56.554411  1216 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:56.556253 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
I20240426 13:22:56.561044 25098 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:56.567248  1254 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.568215  1254 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.570370  1254 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.575685 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.578138 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.579733  1243 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.583331 25589 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.584589 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.587904  1216 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.588398  1216 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.591380  1254 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:56.594082  1221 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.594187 25069 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.591441  1216 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.601097 25589 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.602293 25070 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.602346 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.604854 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.605665  1243 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.605266 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.607007  1216 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.607944  1247 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.608522  1247 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.614324  1254 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:56.615317  1254 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.616290  1216 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.616832  1216 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.619174 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:56.619611  1239 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.620388  1216 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.620152  1239 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.625283  1239 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.619627 25589 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.628037  1254 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.628540 25070 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.630138 25589 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.630578 25069 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.632586 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.635079  1216 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20240426 13:22:56.630934 25589 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:56.630183  1247 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.631997  1239 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.638602  1239 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.640630  1239 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.635591 25070 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.636714 25589 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.643234  1254 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.642378 24849 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 22 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:56.644307 25588 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.650133 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.650768 25622 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 29.
W20240426 13:22:56.654057 25069 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.654898 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.654433 25070 leader_election.cc:334] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.656643  1247 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:56.658033 25069 leader_election.cc:334] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.655297 25589 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.659158 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.662488 25070 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.660226  1247 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:56.663913 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.665200  1216 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20240426 13:22:56.665539  1216 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.665861  1216 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Advancing to term 30
W20240426 13:22:56.668354 25589 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.687572  1216 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.693343  1216 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.693953 25070 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.694250 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:56.694758 25622 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Advancing to term 30
I20240426 13:22:56.702903 25622 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 30.
I20240426 13:22:56.703823 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.708243  1216 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Leader election won for term 30
I20240426 13:22:56.709151  1216 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:56.710023  1216 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.724642 25589 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:56.726482 25589 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.734531 24849 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 29 to 30, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 30 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:56.747210 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:56.748754 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.749773 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.750876 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.751209  1216 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.751678  1216 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.754127 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.754967 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.756019 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.756592 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.757057 25069 leader_election.cc:334] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.757778 25070 leader_election.cc:334] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.758136 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.758833  1244 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:56.758955  1216 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.759385 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.761155 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.762290 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.765133 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.765952 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.767021 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.769874 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.770661 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.771369 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.774078 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.774868 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.775974 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.778702 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.779541 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.780680 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.783450 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.784291 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.784925 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.787537 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.788393 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.789238 23796 pending_rounds.cc:62] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Trying to abort 1 pending ops.
I20240426 13:22:56.789522 23796 pending_rounds.cc:68] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting op as it isn't in flight: id { term: 30 index: 20 } timestamp: 7021108333413793792 op_type: NO_OP noop_request { }
I20240426 13:22:56.790079 23796 raft_consensus.cc:2866] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:56.790421 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.793294 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.794082 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.794670 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.797164 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.797927 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.798954 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.801525 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.802332 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.803375 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.805912 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.806730 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.807500 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.810153 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.810971 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.812197 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.814982 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.815802 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.815820 25589 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.816713  1216 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.817168 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.817236  1216 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.820158 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.821007 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.822271 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.822652  1262 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.823210  1262 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.825038 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.825898 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.825960 25070 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.819113  1216 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.827217 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.827646 25069 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.828142 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.826036  1262 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.829689  1262 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20240426 13:22:56.829933 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.830907 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.831785 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.832178 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.832808 25622 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:22:56.833653 25070 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.834487 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:22:56.837244 25069 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.837716 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.838258  1262 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.835398 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.844473 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.846616 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.847153 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.847976 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.849787 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.850338 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.850812 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.853302 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.854558 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.855330 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.855660 25589 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:56.856420 25589 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.857533 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.858100 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.858857 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.860785 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.861291 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.861771 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.863766 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.864322 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.864807 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.866614 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.867153 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.867916 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.869634 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.870162 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.870774 23796 pending_rounds.cc:62] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Trying to abort 1 pending ops.
I20240426 13:22:56.870941 23796 pending_rounds.cc:68] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting op as it isn't in flight: id { term: 21 index: 18 } timestamp: 7021108332242690048 op_type: NO_OP noop_request { }
I20240426 13:22:56.871217 23796 raft_consensus.cc:2866] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:56.871462 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.873610 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.874126 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.874585 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.876451 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.877002 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.877727 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.880196 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.880861 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.881839 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.883878 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.884479 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.885257 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.885825  1262 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.886353  1262 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.887558 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.888343 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.888523  1262 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.889385 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.892339 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.893031 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.892733 25069 leader_election.cc:334] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.893716 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.894861 25070 leader_election.cc:334] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.895372 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.896137  1262 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.896346 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.898370 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.899389 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.901801 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.902473 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.902580  1262 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.903525 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.903216  1262 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.905759  1262 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.906302 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.908224 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.911845 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.914543 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:22:56.913843 25069 leader_election.cc:334] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.915330 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.916013 25070 leader_election.cc:334] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.916271 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.916462 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.917060 25070 raft_consensus.cc:2670] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240426 13:22:56.918644 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.919226 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.919768 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.921821 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.922348 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.923206 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.925208 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.925707 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.926244 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.928885 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.929584 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.930142 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.931579 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.932729 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.933305 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.934007 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.936357 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.936978 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.937836 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.939868 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.940395 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.940932 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.942714 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.943147 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.943704 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.945432 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.945874 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.946270 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.956544 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.009320 25589 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.011494 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.100375 25589 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.131366 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.158466 25589 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.170512 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.173583 25589 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.319000 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.324774 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.325039 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.330214 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.330530 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.334776 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.335296 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.343670 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.343924 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.370982 25589 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.441947 25589 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.481952 25589 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.489442 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.508494 25589 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.509524 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.527784 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.626454 25589 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.640420 25589 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.667951 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.682479 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.703002 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:57.708134 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.796669 25589 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:57.803711  1258 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:57.804320  1258 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:57.806285  1258 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:57.810668 25589 leader_election.cc:334] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.811316 25589 leader_election.cc:334] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:57.813215 25589 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:57.813874  1254 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:57.816398 25589 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.821712 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240426 13:22:58.147639  1258 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.148284  1258 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.148903  1257 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.149438  1257 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.150650  1258 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.151350  1257 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.154484  1257 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.157277  1270 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.156983  1257 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.159183  1270 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.160490  1257 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.163852  1270 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:58.166326  1257 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.169032  1257 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.167595  1254 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.171701  1254 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.173908  1254 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.176744  1257 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.184625 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:58.185174 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
W20240426 13:22:58.185348 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:58.186256 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:58.186908 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:58.187532 25589 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.188019 25589 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.188745 25589 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.189304 25588 leader_election.cc:334] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.189767 25588 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.189996 25589 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.190153 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.190469 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.190943  1257 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20240426 13:22:58.191097 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:58.191183  1254 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:58.191810 25588 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.192227 25589 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.192358  1254 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.192664 25589 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.192924  1254 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.193389  1272 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:58.193728 25589 leader_election.cc:334] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.194134 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.194911  1254 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.195014 25589 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.195513  1254 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:58.195528 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.196524  1254 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:58.198405 25588 leader_election.cc:334] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.200345 25589 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.201087 25589 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.201506 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.202234  1254 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:22:58.204454 25589 leader_election.cc:334] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.204887 25589 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.205664  1254 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:58.248847  1272 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.249392  1272 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.251585  1272 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.253271  1272 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.253767  1272 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.255791  1272 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:58.256992 25589 leader_election.cc:334] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.257680 25589 leader_election.cc:334] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.258119 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.258833  1272 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:58.258967  1254 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.259649  1254 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:58.261215 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:58.261754  1254 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:58.261844 25589 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.263458 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.264086  1254 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:58.265770 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.268257 25589 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.268675 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.269403  1254 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20240426 13:22:58.271042 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:58.283793 23796 tablet_server.cc:178] TabletServer@127.23.61.4:0 shutting down...
I20240426 13:22:58.317675 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:58.318392 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.319039 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.319576 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.321659 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.322213 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.322695 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.324673 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.325186 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.325676 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.327610 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.328176 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.328614 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.330657 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.331240 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.332337 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.333112  1254 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.333631  1254 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.335448 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.335736  1254 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.336309 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.336846  1254 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.337317  1254 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.338933  1254 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.339082 25588 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:58.339821 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.340462 25589 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.340806 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.341387  1272 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:58.342209 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.342834 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:58.343330 25589 leader_election.cc:334] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:58.343801 25589 leader_election.cc:334] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.343923 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.344146 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:58.344630 25589 raft_consensus.cc:2670] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240426 13:22:58.346222 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.347066 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.347680 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.350546 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.351352 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.352447 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.355149 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.355926 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.356726 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.359433 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.360189 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.361217 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.363968 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.364799 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.365531 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.368214 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.368947 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.369525 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.372153 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.372927 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.374099 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.376865 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.377557 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.378528 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.381074 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.381829 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.382802 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.385376 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.386087 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.386551 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.388743  1272 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.389185  1272 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.389492 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.390343 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.391423 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.390753  1272 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.394419 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
W20240426 13:22:58.394776 25589 leader_election.cc:334] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.395150 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.395741 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.396235 25589 leader_election.cc:334] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.396564 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.397181  1272 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:58.398433 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.399178 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.400120 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.402710 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.403416 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.404373 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.406716 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.407385 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.407979 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.410519 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.411149 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.412117 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.414573 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.415148 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.416069 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.418408 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.419036 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.419595 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.422169 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.422847 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.423807 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.426101 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.426728 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.427193 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.429136 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.429617 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Raft consensus shutting down.
W20240426 13:22:58.430234 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.430481 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.433560 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.434307 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.435200 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.435815 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:58.436089 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.437379 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.437983 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.438897 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.439973 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.440261 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.441062 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.441926 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.442505 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.444370 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.444593 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.445053 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.445633 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.446589 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.448781 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.448868 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
W20240426 13:22:58.449049 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.449515 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.450374 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.452481 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.452970 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.453696 23796 pending_rounds.cc:62] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Trying to abort 1 pending ops.
I20240426 13:22:58.453860 23796 pending_rounds.cc:68] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting op as it isn't in flight: id { term: 22 index: 18 } timestamp: 7021108333025619968 op_type: NO_OP noop_request { }
I20240426 13:22:58.454125 23796 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:58.454375 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.456096 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.456586 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.456985 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.458719 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.459223 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.460072 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.462093 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.462554 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.463263 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.465066 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.465602 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.465996 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.467787 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.468291 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.469009 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.470695 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.471109 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.471529 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.748260 23796 tablet_server.cc:194] TabletServer@127.23.61.4:0 shutdown complete.
I20240426 13:22:58.804746 23796 master.cc:537] Master@127.23.61.62:34153 shutting down...
I20240426 13:22:58.828001 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.828554 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.828860 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: stopping tablet replica
W20240426 13:22:58.836789 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08000082e0 after lost signal to thread 23799
W20240426 13:22:58.837991 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080032f1c0 after lost signal to thread 24888
W20240426 13:22:58.839412 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080040c9c0 after lost signal to thread 25650
I20240426 13:22:59.341562 23796 master.cc:552] Master@127.23.61.62:34153 shutdown complete.
I20240426 13:22:59.379818 23796 test_util.cc:175] -----------------------------------------------
I20240426 13:22:59.380008 23796 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0

[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 13:17:37.835705 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:39733
I20240426 13:17:37.837103 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 13:17:37.838186 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:37.854764 23805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:37.854813 23802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:37.854805 23803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:37.856245 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:38.949949 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:38.950196 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:38.950349 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137458950334 us; error 0 us; skew 500 ppm
I20240426 13:17:38.951161 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:38.956547 23796 webserver.cc:460] Webserver started at http://127.23.61.62:37521/ using document root <none> and password file <none>
I20240426 13:17:38.957419 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:38.957612 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:38.957986 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:38.964125 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "5631be9661c5413097ecac9e9e7835e5"
format_stamp: "Formatted at 2024-04-26 13:17:38 on dist-test-slave-jx77"
I20240426 13:17:38.971596 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.000s
I20240426 13:17:38.976673 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:38.977733 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
I20240426 13:17:38.978039 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "5631be9661c5413097ecac9e9e7835e5"
format_stamp: "Formatted at 2024-04-26 13:17:38 on dist-test-slave-jx77"
I20240426 13:17:38.978356 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.042387 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:17:39.043839 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 13:17:39.044274 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.112521 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:39733
I20240426 13:17:39.112605 23862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:39733 every 8 connection(s)
I20240426 13:17:39.119544 23863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:39.142756 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Bootstrap starting.
I20240426 13:17:39.148125 23863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:39.149859 23863 log.cc:824] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Log is configured to *not* fsync() on all Append() calls
I20240426 13:17:39.154168 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: No bootstrap required, opened a new log
I20240426 13:17:39.175776 23863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.176342 23863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:39.176538 23863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5631be9661c5413097ecac9e9e7835e5, State: Initialized, Role: FOLLOWER
I20240426 13:17:39.177264 23863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.177742 23863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:17:39.177973 23863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:17:39.178249 23863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:39.181891 23863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.182507 23863 leader_election.cc:302] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5631be9661c5413097ecac9e9e7835e5; no voters: 
I20240426 13:17:39.184083 23863 leader_election.cc:288] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 13:17:39.184415 23866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:39.186975 23866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5631be9661c5413097ecac9e9e7835e5, State: Running, Role: LEADER
I20240426 13:17:39.187777 23866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.188818 23863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:17:39.198256 23867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5631be9661c5413097ecac9e9e7835e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER } }
I20240426 13:17:39.198633 23868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5631be9661c5413097ecac9e9e7835e5. Latest consensus state: current_term: 1 leader_uuid: "5631be9661c5413097ecac9e9e7835e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER } }
I20240426 13:17:39.199178 23868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: This master's current role is: LEADER
I20240426 13:17:39.199178 23867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: This master's current role is: LEADER
I20240426 13:17:39.202414 23876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:17:39.213723 23876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:17:39.220765 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:17:39.230899 23876 catalog_manager.cc:1347] Generated new cluster ID: 57169e0a2011421f9d4f00a8bad18da1
I20240426 13:17:39.231202 23876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:17:39.246026 23876 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:17:39.248092 23876 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:17:39.270504 23876 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Generated new TSK 0
I20240426 13:17:39.271649 23876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:17:39.287178 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.295161 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.296100 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.302160 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.303112 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.303745 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.303943 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.304131 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459304115 us; error 0 us; skew 500 ppm
I20240426 13:17:39.304713 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.308569 23796 webserver.cc:460] Webserver started at http://127.23.61.1:44903/ using document root <none> and password file <none>
I20240426 13:17:39.309060 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.309281 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.309554 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.310653 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.316445 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240426 13:17:39.320454 23893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.321280 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240426 13:17:39.321568 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.321852 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.361940 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:17:39.363134 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.365703 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.370214 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.370432 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.370733 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.370885 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.472268 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:44541
I20240426 13:17:39.472393 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:44541 every 8 connection(s)
I20240426 13:17:39.478868 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.491952 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.497804 23961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.508445 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.513607 23956 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.514098 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.514106 23956 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.514999 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.515209 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.515383 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459515368 us; error 0 us; skew 500 ppm
I20240426 13:17:39.515380 23956 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
I20240426 13:17:39.516150 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.519349 23828 ts_manager.cc:194] Registered new tserver with Master: cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541)
I20240426 13:17:39.519467 23796 webserver.cc:460] Webserver started at http://127.23.61.2:34327/ using document root <none> and password file <none>
I20240426 13:17:39.520174 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.520429 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.520774 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.522296 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "decbe9ee3d0a47bab14b976ecaf12843"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.522325 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54958
I20240426 13:17:39.528173 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240426 13:17:39.532128 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.532915 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240426 13:17:39.533179 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "decbe9ee3d0a47bab14b976ecaf12843"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.533473 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.558553 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:17:39.559794 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.562222 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.565181 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.565358 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.565609 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.565748 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.661548 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:35101
I20240426 13:17:39.661687 24030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:35101 every 8 connection(s)
I20240426 13:17:39.669883 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.678973 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.682168 24031 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.682664 24031 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.683866 24031 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
W20240426 13:17:39.686048 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.686450 23828 ts_manager.cc:194] Registered new tserver with Master: decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:39.688331 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54970
W20240426 13:17:39.688760 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.693871 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.694659 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.694859 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.695030 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459695011 us; error 0 us; skew 500 ppm
I20240426 13:17:39.695574 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.698576 23796 webserver.cc:460] Webserver started at http://127.23.61.3:40183/ using document root <none> and password file <none>
I20240426 13:17:39.699074 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.699277 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.699532 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.700862 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "20b204c02fe14d9eb754d5a169d43996"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.705523 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20240426 13:17:39.708752 24042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.709534 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20240426 13:17:39.709789 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "20b204c02fe14d9eb754d5a169d43996"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.710047 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.747560 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:17:39.748759 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.750208 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.753605 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.753803 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.754062 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.754205 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:17:39.847970 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:45215
I20240426 13:17:39.848059 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:45215 every 8 connection(s)
I20240426 13:17:39.864207 24105 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.864549 24105 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.865213 24105 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
I20240426 13:17:39.867152 23828 ts_manager.cc:194] Registered new tserver with Master: 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:39.868008 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015310463s
I20240426 13:17:39.868956 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54980
I20240426 13:17:39.872473 23796 test_util.cc:268] Using random seed: -1167830334
I20240426 13:17:39.921759 23828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54988:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20240426 13:17:39.925360 23828 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:17:40.029588 23920 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.030342 23919 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.031277 23920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.031996 23919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.034143 23918 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.035632 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.033249 23921 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.041544 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.046172 23917 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.047506 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.051560 23916 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.054275 23916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.059098 23915 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.061138 23915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.070240 24070 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.070798 24069 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.073320 24070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.084362 24069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.076122 24068 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.086189 24067 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.089030 24065 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.095767 24068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.097096 24067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.081506 24066 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.095208 23914 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.096175 24064 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.111622 23996 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.107662 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.110447 23914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.119814 23996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.111455 24063 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.126462 23995 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.118155 23913 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.126873 24066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.139787 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.142683 24062 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.137437 23994 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.143545 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.132019 23993 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.149688 24123 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.156499 23913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.149662 23992 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.162138 23989 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.163507 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.166716 23988 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.167678 23990 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.171777 23994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.173537 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.170680 24062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.177692 23991 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.189545 23992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.202613 23991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.204478 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:17:40.213791 23875 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:17:40.217510 23993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.221472 24063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.318614 24123 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.349287 24124 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.351624 24125 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.367540 24123 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.368764 24124 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.370522 24123 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.236s	user 0.015s	sys 0.004s
I20240426 13:17:40.374158 24123 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.374966 24123 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.375272 24123 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.375663 24125 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.375979 24123 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.389058 23956 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.389614 24123 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.013s
I20240426 13:17:40.390476 24124 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.390763 24123 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.390906 24124 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.053s	user 0.012s	sys 0.007s
I20240426 13:17:40.396493 24123 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.403947 24124 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.404999 24124 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.405375 24124 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.409324 24125 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.409829 24125 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.078s	user 0.013s	sys 0.023s
I20240426 13:17:40.410676 24124 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.412696 24125 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.413612 24125 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.413952 24125 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.414752 24125 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.422921 24123 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.423394 24123 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.011s
I20240426 13:17:40.423768 24125 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.000s
I20240426 13:17:40.424257 24124 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.003s
I20240426 13:17:40.424533 24031 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.424857 24125 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.428650 24105 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.428897 24124 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.432303 24132 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.432848 24132 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.436134 24123 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.436928 24123 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.437223 24123 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.438850 24125 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.439136 24123 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.441327 24124 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.446094 24132 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.450212 24123 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.002s
I20240426 13:17:40.457346 24124 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.457775 24124 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.005s
I20240426 13:17:40.460738 24125 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.462314 24125 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.009s
I20240426 13:17:40.465607 24123 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.460364 24124 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.468299 24124 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.468461 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.469094 24124 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.470989 24124 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
W20240426 13:17:40.472391 24045 leader_election.cc:341] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.473570 24124 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20240426 13:17:40.473081 24125 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.474434 24124 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.474706 24125 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.476006 24125 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.477352 24123 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
W20240426 13:17:40.480491 23957 tablet.cc:2369] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 13:17:40.477552 24125 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.484915 24124 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.489032 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.489771 24006 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.490212 24125 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20240426 13:17:40.491216 24044 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.491516 24125 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.492426 24132 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.492798 24132 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.493167 24132 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.506583 24125 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.521322 24132 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.525436 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.526082 24132 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
W20240426 13:17:40.526758 24045 leader_election.cc:341] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.525548 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.527477 24006 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.527968 24125 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.528424 24125 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.001s
I20240426 13:17:40.529295 24123 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.529779 24123 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.064s	user 0.036s	sys 0.016s
I20240426 13:17:40.531131 24125 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.531858 24125 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.531881 24006 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.532295 24125 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.532512 24123 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.533264 24044 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.533526 24123 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.533998 24123 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.534035 24124 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.533149 24125 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.534740 24124 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.060s	user 0.021s	sys 0.015s
I20240426 13:17:40.534961 24132 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.535578 24123 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.538118 24123 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:17:40.539057 24123 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.544752 24123 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.545385 24125 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.005s
I20240426 13:17:40.546375 24125 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.549896 24124 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.550650 24124 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.550935 24124 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.551659 24124 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.553805 24125 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.557114 24123 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.557605 24123 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.007s
I20240426 13:17:40.558167 24132 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.558425 24124 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.003s
I20240426 13:17:40.559296 24124 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.558959 24132 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.560410 24123 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.561136 24123 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.561476 24123 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.562572 24123 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.565017 24123 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:17:40.566376 24123 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.568895 24124 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.573118 24123 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.575631 24125 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.576145 24125 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.005s
I20240426 13:17:40.578909 24125 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.579644 24125 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.580008 24125 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.581367 24124 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.580786 24125 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.581786 24124 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.006s
I20240426 13:17:40.580941 23826 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.584755 24124 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.585563 24124 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.585932 24124 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.588095 24132 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.588629 24132 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.590744 24125 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.004s
I20240426 13:17:40.591078 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.591629 24125 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
W20240426 13:17:40.592309 24045 leader_election.cc:341] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.593173 24132 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.586966 24124 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.595175 24124 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20240426 13:17:40.595266 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.595860 24006 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.596057 24124 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.597549 24044 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.598392 24125 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.598493 24142 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.598940 24142 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.599238 24142 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.604692 24123 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.605132 24123 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.007s
I20240426 13:17:40.605221 24142 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.608196 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
W20240426 13:17:40.609408 24045 leader_election.cc:341] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.610217 24142 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.611091 24124 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.611184 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.611757 24006 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.617350 24123 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.618044 24123 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.618299 24123 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.618966 24123 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.619400 24006 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.621730 24123 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20240426 13:17:40.625308 24123 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.625814 24044 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.627259 24142 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.627696 24125 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.628218 24125 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.037s	user 0.006s	sys 0.009s
I20240426 13:17:40.628124 24142 raft_consensus.cc:686] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.629668 24142 consensus_queue.cc:237] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.631338 24123 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.631227 24125 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.631976 24125 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.637549 24125 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.640313 24125 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.642741 24124 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.644747 24124 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.049s	user 0.014s	sys 0.021s
I20240426 13:17:40.647562 24124 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.644309 24125 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.002s
I20240426 13:17:40.648147 24123 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.649204 24124 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.649338 24123 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.005s
I20240426 13:17:40.650173 24125 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.652356 24123 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.653203 24123 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.653538 24123 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.654294 24123 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.656610 24123 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20240426 13:17:40.657042 24125 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.657387 24124 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.659981 24123 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.665285 24124 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.667920 23826 catalog_manager.cc:5484] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:40.671897 24124 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.001s
I20240426 13:17:40.673027 24124 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.678256 24125 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.678853 24125 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.029s	user 0.002s	sys 0.012s
I20240426 13:17:40.679518 24124 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.679759 24142 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.680233 24142 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.673522 24123 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.682191 24142 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.683271 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.684773 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
W20240426 13:17:40.686892 24045 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.691648 24123 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.691725 24124 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.692204 24123 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.005s
W20240426 13:17:40.692238 24044 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.692278 24124 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
I20240426 13:17:40.692860 24044 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996; no voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.693710 24142 raft_consensus.cc:2726] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.694716 24128 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.695091 24128 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.695683 24125 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696139 24123 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696498 24125 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.696573 24124 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696892 24125 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.697432 24128 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.704943 24124 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.705519 24124 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.706388 24124 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.706038 24125 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.709203 24124 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.008s
I20240426 13:17:40.709662 24142 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.710098 24142 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.711987 24142 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.713266 24124 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.713191 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.713949 24006 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.716476 24044 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.717259 24142 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.717613 24142 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.717948 24142 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.718115 24125 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20240426 13:17:40.719221 24125 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.696862 24123 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.722208 24123 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.723335 24123 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.724625 24142 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.726656 24142 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.728320 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.728899 24006 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.730126 24124 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.726730 24123 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.004s
I20240426 13:17:40.731741 24125 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.735697 24123 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.742617 24006 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.743793 24044 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.744581 24142 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.745050 24142 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.745836 24142 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.750820 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.751441 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.752692 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.753249 23931 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.758877 24123 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.764199 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.764541 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.765303 24006 raft_consensus.cc:2379] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.769855 24131 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.770892 24131 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.778064 24131 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.781931 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.784618 23895 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.786084 24128 raft_consensus.cc:2726] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.785290 23828 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:40.790382 24128 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.790961 24128 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.793308 24128 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.795328 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.796005 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.796365 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.797003 24006 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.797137 23895 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:40.797964 24128 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.798327 24128 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.798671 24128 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.799355 24131 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.799945 24131 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.805765 24131 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.806064 24128 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.809316 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.809512 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.809922 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.810245 24006 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.817494 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.818588 23895 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:40.808163 24128 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.821492 24128 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.821935 24128 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:40.822804 24128 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.824688 24125 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.825289 24125 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.106s	user 0.021s	sys 0.010s
I20240426 13:17:40.819589 24006 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.831277 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.833611 24079 raft_consensus.cc:2379] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate decbe9ee3d0a47bab14b976ecaf12843 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:40.825771 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.833878 24125 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.834858 24080 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.834925 24125 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.835625 24125 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.836572 23970 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.836503 24125 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.837785 24131 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.838268 24131 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.841248 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.841926 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.842932 24166 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:17:40.843544 23971 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.838713 24131 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.843581 24166 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.844468 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.845965 24166 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.847358 24170 raft_consensus.cc:2726] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.847841 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.850073 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.850849 24006 raft_consensus.cc:2379] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.851436 24124 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.851894 24124 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.139s	user 0.032s	sys 0.047s
I20240426 13:17:40.852865 24125 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.008s
I20240426 13:17:40.853905 24125 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.854795 24124 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.855518 24124 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.855814 24124 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.856549 24124 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.859149 24124 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:17:40.860430 24125 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.861549 24123 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.861414 23895 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.861980 24123 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.127s	user 0.013s	sys 0.000s
I20240426 13:17:40.862337 24167 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.867548 24123 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.868283 24123 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.868557 24123 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.869130 24123 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.870573 24167 raft_consensus.cc:2726] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.871124 24123 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20240426 13:17:40.872294 24124 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.875572 24131 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.871971 24123 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.878885 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.879729 23930 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.880357 24128 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.880774 24128 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.882715 24128 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.884083 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.884675 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.885305 24131 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.885051 24123 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.886083 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.885502 24080 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.886776 24079 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.888083 23895 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:40.888926 24166 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240426 13:17:40.889070 23895 leader_election.cc:341] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.889286 24166 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.889642 24166 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.893066 23930 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.879007 23826 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.893752 24166 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.894413 23971 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.895148 24131 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.895537 24131 raft_consensus.cc:686] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:40.896327 24131 consensus_queue.cc:237] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.900121 24166 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.901618 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.904850 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.905578 24079 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.909654 24123 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.910115 24123 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.000s
I20240426 13:17:40.912734 24123 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.913445 24123 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.913386 24131 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.913830 24123 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.914012 24124 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.914125 24131 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.914639 24123 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.915489 24080 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.916134 24131 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.916909 24123 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
W20240426 13:17:40.920120 23895 leader_election.cc:341] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.920874 24166 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.922049 24166 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.919203 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.929186 24166 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.929626 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.911305 23827 catalog_manager.cc:5484] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 0 to 1, leader changed from <none> to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.932754 24078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.933616 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.937009 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.937104 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.937363 23971 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.937870 24006 raft_consensus.cc:2379] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.938138 24125 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.938501 24125 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.085s	user 0.035s	sys 0.019s
I20240426 13:17:40.939337 24131 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.939705 24131 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.940058 24131 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.940725 24125 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.940066 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.941354 24125 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.941659 24125 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.942044 24166 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.942232 24125 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.944191 24125 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20240426 13:17:40.944962 24125 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.950417 24125 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.946703 24131 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.952248 24166 raft_consensus.cc:2726] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.954147 24079 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.955143 23895 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.956998 24124 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.957202 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.957520 24124 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.098s	user 0.012s	sys 0.024s
I20240426 13:17:40.957816 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.957676 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.959199 24079 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.960283 24124 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.960930 24124 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.961175 24124 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.956167 24131 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.961752 24124 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.964941 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.967144 23971 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.967911 24131 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.965364 24166 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.968281 24125 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.968524 24128 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.968746 24166 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:40.968868 24125 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.006s
I20240426 13:17:40.968992 24128 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.969594 24166 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.970919 24128 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.972637 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.972734 24131 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:40.973155 24124 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.005s
I20240426 13:17:40.972812 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.971328 24125 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.973938 24125 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.974231 24125 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
W20240426 13:17:40.974380 23895 leader_election.cc:341] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.974623 24131 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.974895 24125 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.975782 24006 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.978240 24124 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.978781 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:40.979754 24167 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.980211 24167 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.980522 24167 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.983834 24124 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.965080 24079 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.987303 24167 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.990113 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.990710 24006 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.991040 23826 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 0 to 1, leader changed from <none> to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.997802 24006 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.998854 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:40.999657 24128 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:41.007465 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
W20240426 13:17:41.008878 23895 leader_election.cc:341] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:41.008339 24125 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.001s
I20240426 13:17:41.008410 24124 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:41.010556 24124 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.006s
I20240426 13:17:41.008908 24167 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.013924 24124 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.019982 24124 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:41.014430 24131 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.020349 24124 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:41.020746 24131 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.021150 24124 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.022763 24131 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.014159 24128 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.023754 24128 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.024312 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:41.025086 24079 raft_consensus.cc:2379] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate decbe9ee3d0a47bab14b976ecaf12843 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:41.025153 24142 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.028018 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.030153 23971 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.031363 24131 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.032721 24124 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.003s
I20240426 13:17:41.032577 23827 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.037680 24131 raft_consensus.cc:2726] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:41.056020 24142 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.056169 23826 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.073875 24142 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.087780 24132 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.147023 24187 raft_consensus.cc:482] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.147588 24187 raft_consensus.cc:504] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.150357 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.150527 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:41.150959 24187 leader_election.cc:288] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.151036 23930 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:41.151132 24006 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:41.152173 24045 leader_election.cc:302] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:41.152927 24187 raft_consensus.cc:2781] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:41.153264 24187 raft_consensus.cc:482] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:41.153584 24187 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.159864 24187 raft_consensus.cc:504] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.161731 24187 leader_election.cc:288] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.162462 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.162722 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.162922 23930 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.163324 24006 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.166985 23930 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:41.167841 24045 leader_election.cc:302] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:41.168490 24187 raft_consensus.cc:2781] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:41.168870 24187 raft_consensus.cc:686] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.169364 24006 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:41.169711 24187 consensus_queue.cc:237] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.177804 23827 catalog_manager.cc:5484] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:41.201128 24187 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.201599 24187 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.203087 24187 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.203963 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.204090 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:41.204876 24006 raft_consensus.cc:2379] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20b204c02fe14d9eb754d5a169d43996 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:41.205694 24044 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996; no voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.206307 24187 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.212432 24187 raft_consensus.cc:2726] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:41.322573 24131 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.323727 24187 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.341476 24187 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.341697 24170 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.346462 24166 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.370002 24179 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.370404 24170 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.382313 24131 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.420261 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.420964 24079 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.428406 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.428953 24079 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.441273 24006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:43458
I20240426 13:17:41.441722 24006 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.453980 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:41.454629 23930 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.458812 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.459328 24079 raft_consensus.cc:595] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.465900 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:41.466481 23930 raft_consensus.cc:595] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.485009 24187 raft_consensus.cc:979] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.485445 24140 raft_consensus.cc:1067] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.486796 24006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:41.487227 24006 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.487534 24006 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.491685 24006 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.493429 24006 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.493885 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.494012 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.494437 23930 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.494590 24079 raft_consensus.cc:3032] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.494939 24079 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.495644 24079 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.497144 24079 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.498561 23930 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.499471 23971 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:41.500082 24131 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.500453 24131 raft_consensus.cc:686] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.501082 24131 consensus_queue.cc:237] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.503502 24079 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.507874 23825 catalog_manager.cc:5484] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.521039 24166 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.531581 24166 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.532006 24128 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.547852 24140 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.548314 24140 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.553309 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34980
I20240426 13:17:41.553867 23930 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.554216 23930 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.566092 23930 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.567726 24166 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.568600 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.569327 24079 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.569542 23930 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.569710 24079 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.570920 24079 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.572525 24079 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.573963 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.574579 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.579485 24079 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.581022 24005 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.580852 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:41.582432 24166 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.582906 24166 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.583678 24166 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.589185 24166 raft_consensus.cc:979] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.589602 24167 raft_consensus.cc:1067] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.591127 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:41.591643 24005 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.591984 24005 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.591851 23825 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.598505 24005 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.601707 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.601914 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.602298 23930 raft_consensus.cc:3032] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.602423 24005 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.602632 23930 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.602725 24079 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.603399 23930 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.604626 23930 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.609084 24079 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.610209 23970 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:41.610921 23930 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.611100 24131 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.611627 24131 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.612394 24131 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.617591 24142 consensus_queue.cc:1035] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.619428 23825 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.631590 24187 consensus_queue.cc:1035] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.817775 24131 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.818275 24169 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.819852 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:41.820490 23930 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.820853 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.887712 23930 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.889559 23930 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.890774 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.891290 24005 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.891608 24005 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.891965 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.892534 24079 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.892396 24005 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.893626 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.952190 24142 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.952637 24187 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.954231 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34980
I20240426 13:17:41.954689 23930 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.954975 23930 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.992882 24004 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:41.993358 24005 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.994407 23930 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.995290 24128 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:41.996317 23930 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.996408 24166 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.996727 24079 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.996996 24166 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.997900 23895 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:41.998672 24166 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.000012 24166 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.000352 24004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:42.000954 24004 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:42.001327 24004 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.004118 24079 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.004739 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:42.005331 24080 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:42.005681 24080 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:42.006449 24080 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.001279 24166 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.007676 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.006831 24169 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:17:42.009472 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:42.010073 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.014720 23930 raft_consensus.cc:1261] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.016172 24079 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.017751 24179 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.018899 24170 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.024075 23931 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.026100 24169 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.029770 23825 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.034168 24079 raft_consensus.cc:1261] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.035902 24169 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.093554 24080 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:42.093595 24004 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.098104 23895 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:42.098181 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:42.099027 23931 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:42.099172 24166 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.099416 23931 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.099730 24166 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.099776 24004 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:42.100734 24166 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.101576 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:42.100250 23931 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.102166 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.102843 23931 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.107352 23825 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.192867 24005 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:42.193125 23931 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:42.192871 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:42.194376 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:42.195351 24169 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.195837 24169 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:42.196569 24169 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.202766 23825 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.411480 24005 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.412662 24179 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.420836 24080 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.424335 24166 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.663064 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.664139 24170 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.671391 24080 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.673247 24170 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.682006 24080 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.683010 24166 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.692304 24005 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.693842 24179 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:44.490451 23796 auto_leader_rebalancer-test.cc:261] The leader distribution is 
I20240426 13:17:44.491189 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:44.510504 23931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:50698
I20240426 13:17:44.510989 23931 raft_consensus.cc:595] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:44.520948 24004 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:44.521549 24004 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:44.522707 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 13:17:44.523098 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:44.645644 24169 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:44.646091 24215 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:44.647428 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:44.647872 24080 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:44.648181 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.652292 24080 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.653923 24080 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:44.654680 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:44.655253 23931 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.655982 24004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:44.656558 24004 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:44.656889 24004 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:44.657589 24004 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.658797 24004 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.661572 23931 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 3.
I20240426 13:17:44.662607 24045 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:44.663331 24140 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:44.664542 24004 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 3.
I20240426 13:17:44.664516 24140 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:44.665427 24140 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.671753 23828 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 2 to 3, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:45.008005 24179 raft_consensus.cc:979] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:45.008391 24225 raft_consensus.cc:1067] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:45.009616 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:44396
I20240426 13:17:45.010028 24080 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:45.010368 24080 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.014817 24080 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.016273 24080 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:45.016971 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:45.017172 24004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:45.017434 23931 raft_consensus.cc:3032] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:45.017670 24004 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.017730 23931 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:45.018440 23931 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.019361 23931 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.022750 24004 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:45.023609 24044 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:45.024402 24140 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:45.024796 23931 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:45.024853 24140 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:45.025806 24140 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.034741 23828 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:45.123018 24004 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:45.124135 24140 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:45.131906 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:45.133278 24219 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:45.523866 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:45.524454 24004 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:45.525841 24140 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:45.531371 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:45.531926 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:45.540438 23931 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:45.541671 24140 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:46.532848 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:46.538168 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:46.538589 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:47.539420 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:47.544373 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:47.544760 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:48.545725 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:48.551060 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:48.551445 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:49.552165 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:49.556962 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:49.557371 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:50.558069 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:50.564134 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:50.564656 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:51.565469 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:51.570698 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:51.571151 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:52.571970 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:52.576527 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:52.576835 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:53.577527 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:53.582331 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:53.582777 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:54.602828 23796 auto_leader_rebalancer-test.cc:280] The leader distribution is 
I20240426 13:17:54.606851 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.607470 23930 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.611738 24080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:54.612191 24080 raft_consensus.cc:595] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.615906 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.616338 23930 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.619841 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.620218 23930 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.623816 24080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:54.624223 24080 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.653193 24246 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.653652 24248 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.654886 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.655336 24005 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.655706 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.661381 24005 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.662964 24005 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.663605 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.663826 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.664546 24080 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.664501 23930 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.665051 23930 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.665647 23930 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.666494 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.668794 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.669734 23970 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:54.670396 24215 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.670598 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.671300 24215 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.672159 24215 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.678185 23825 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.715777 24140 raft_consensus.cc:979] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.716389 24256 raft_consensus.cc:1067] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.717639 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:54.718068 24005 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.718322 24005 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.722512 24005 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.723902 24005 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.724635 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.724731 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.725143 23930 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.725210 24080 raft_consensus.cc:3032] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.725571 24080 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:54.726256 24080 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.727332 24080 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.730640 23930 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.731540 23971 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:54.732156 24215 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.732348 24080 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.732578 24215 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.733283 24215 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.739053 23825 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.771231 24246 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.771636 24248 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.772786 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.773219 24005 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.773486 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.777645 24005 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.778982 24005 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.779702 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.779758 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.780265 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.780370 23930 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.780675 23930 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.781149 23930 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.781966 23930 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.784163 24080 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.784881 23970 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:54.785588 24215 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.785912 24215 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.786190 23930 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.786605 24215 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.792882 23825 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.890141 24246 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.890628 24248 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.891773 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.892216 24005 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.892454 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.896148 24005 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.897382 24005 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541)
I20240426 13:17:54.898182 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.898360 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.898715 24080 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.898830 23930 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.899152 23930 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.899818 23930 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.900913 23930 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.904460 24080 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.905472 23970 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:54.906147 24215 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.906276 23930 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.906589 24215 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.907301 24215 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.913605 23825 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:54.966085 24255 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.966459 24256 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.967638 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
 from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:54.968047 24005 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.968286 24005 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.972147 24005 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.973471 24005 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 4 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.974351 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.974412 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.974912 24080 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:54.974941 23930 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.975261 24080 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:54.975876 24080 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.976727 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.978956 23930 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 4.
I20240426 13:17:54.979806 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:54.980526 24215 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:54.980854 24215 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.981485 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 4.
I20240426 13:17:54.981526 24215 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.987355 23825 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 3 to 4, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:55.156229 23930 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.157480 24249 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.166864 24080 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.168776 24249 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.238744 24080 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.239775 24249 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.247682 23930 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.249034 24249 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.259029 23930 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.260160 24249 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.268187 24080 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.269809 24249 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:17:55.330651 24080 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.331707 24249 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.341310 23930 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.342682 24272 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.445375 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:55.446470 24272 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:55.453888 24080 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:55.455201 24272 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:57.651772 23796 auto_leader_rebalancer-test.cc:295] The leader distribution is 
I20240426 13:17:57.652379 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:57.658703 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.659121 24005 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:57.661029 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.661392 24005 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.663005 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.663352 24005 raft_consensus.cc:595] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:57.665069 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.665438 24005 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.666989 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.667336 24005 raft_consensus.cc:595] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.668148 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:17:57.668519 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:57.823755 24277 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:57.824172 24215 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:57.825237 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:57.825642 24080 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:57.825877 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.829583 24080 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.831228 24080 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 4 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:57.831979 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:57.832021 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:57.832604 24005 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:57.832901 24005 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:57.833248 23931 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.833415 24005 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.834266 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.837296 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 4.
I20240426 13:17:57.837980 24005 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 4.
I20240426 13:17:57.838317 24045 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:57.839063 24256 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:57.839959 24256 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:57.840703 24256 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.846005 23828 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:57.970664 24285 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:57.971367 24215 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:57.972764 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:57.973305 23931 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:57.973623 23931 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.978976 23931 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:57.981030 23931 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:57.982265 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:57.982609 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:57.983199 24005 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:57.983253 24079 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.983597 24005 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:57.984512 24005 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:57.985702 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.990443 24079 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:57.991253 24080 raft_consensus.cc:1226] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Rejecting Update request from peer decbe9ee3d0a47bab14b976ecaf12843 for earlier term 3. Current term is 4. Ops: []
I20240426 13:17:57.991881 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:57.992527 24005 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:57.994555 24286 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:57.995898 24286 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:57.997088 24286 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:58.006286 23828 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.063885 24285 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.064328 24215 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.065466 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:58.065855 23931 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.066088 23931 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.069998 23931 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.071161 23931 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:58.071873 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.072144 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:58.072353 24005 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:58.072633 24005 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.072652 24080 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.073293 24005 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.074258 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.078260 24005 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:58.078986 23895 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:58.079006 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:58.079671 24286 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:58.080137 24286 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:58.080776 24286 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.086335 23828 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.100432 24285 raft_consensus.cc:979] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.100759 24215 raft_consensus.cc:1067] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.101894 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
 from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:58.102252 23931 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.102471 23931 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.106499 23931 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.107430 24285 raft_consensus.cc:979] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:58.107889 24215 raft_consensus.cc:1067] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:58.107957 23931 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 3 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:58.108556 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.108862 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:58.109091 24005 raft_consensus.cc:3032] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:58.109458 24005 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.109405 24080 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.109376 24079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
 from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:58.110347 24079 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.110792 24079 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.110612 24005 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.111886 24005 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.115483 24080 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 3.
I20240426 13:17:58.116228 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: 
I20240426 13:17:58.116760 24286 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:58.117153 24286 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:58.117688 24005 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 3.
I20240426 13:17:58.116859 24079 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.117902 24286 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.119668 24079 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:58.119921 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:58.120193 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.120644 23931 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.120995 24005 raft_consensus.cc:3032] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:58.121238 24005 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.121755 24005 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.122679 24005 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.126248 24005 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:58.127202 24044 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: 
I20240426 13:17:58.127858 23931 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:58.131011 24256 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:58.131314 24256 raft_consensus.cc:686] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:58.132089 24256 consensus_queue.cc:237] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.133785 23828 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 2 to 3, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 3 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.139058 23825 catalog_manager.cc:5484] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 1 to 2, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:58.404616 24079 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.406026 24286 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.415146 24005 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.416540 24286 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.431103 24005 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.432204 24256 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.440541 23931 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.441776 24256 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.489435 24079 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.490466 24286 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.498703 24005 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.499966 24286 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.573761 24079 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:58.574909 24301 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:58.580253 23931 raft_consensus.cc:1261] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:58.583189 24256 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:58.584244 24005 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:58.587536 24286 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:58.590664 24004 raft_consensus.cc:1261] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:58.593616 24278 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:17:58.669427 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:58.674559 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:58.674939 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:59.675603 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:59.680959 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:59.681352 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:00.682124 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:00.686605 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:00.686975 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:01.687716 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:01.692402 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:01.692787 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:02.775274 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:02.780146 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:02.780464 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 13:18:03.358970 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800171060 after lost signal to thread 23799
W20240426 13:18:03.359694 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800171740 after lost signal to thread 23862
W20240426 13:18:03.360456 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800080960 after lost signal to thread 23955
W20240426 13:18:03.361138 24101 debug-util.cc:398] Leaking SignalData structure 0x7b08001f25a0 after lost signal to thread 24030
W20240426 13:18:03.361752 24101 debug-util.cc:398] Leaking SignalData structure 0x7b080011efc0 after lost signal to thread 24104
I20240426 13:18:03.781195 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:03.786418 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:03.786772 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:04.787554 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:04.792912 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:04.793283 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:05.794121 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:05.798740 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:05.799149 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:06.799944 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:06.804607 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:06.805001 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:07.824869 23796 auto_leader_rebalancer-test.cc:306] The leader distribution is 
I20240426 13:18:07.825474 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
I20240426 13:18:07.847337 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:07.848044 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.848973 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.849563 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.852523 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.853229 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.853577 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.854979 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.855384 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.855978 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.857324 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.857712 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.858039 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.859324 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.859716 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.860297 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.861517 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.861937 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.862495 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.863826 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.864312 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.864672 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.866003 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.866372 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.866711 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.868052 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.868476 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.868826 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus is shut down!
W20240426 13:18:07.975165 24045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111)
W20240426 13:18:07.978703 24045 consensus_peers.cc:485] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:07.979096 24045 consensus_peers.cc:485] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:07.985519 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
W20240426 13:18:08.004683 24045 consensus_peers.cc:485] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:08.008353 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
I20240426 13:18:08.033958 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:08.034828 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.035974 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.037333 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.040086 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.040917 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.041491 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.043419 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.043965 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.044473 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.046082 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.046487 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.046936 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.048570 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.049010 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.049423 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.050899 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.051326 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.052361 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.053972 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.054364 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.055115 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.056811 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.057260 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.057682 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.059291 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.059671 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.060104 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.092536 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:18:08.110172 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
W20240426 13:18:08.135170 24044 consensus_peers.cc:485] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 -> Peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Couldn't send request to peer decbe9ee3d0a47bab14b976ecaf12843. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:35101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:08.137109 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:08.137778 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.138432 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.139289 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.141184 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.141639 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.142058 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.143770 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.144306 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.144766 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.146273 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.146689 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.147076 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.148838 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.149240 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.149986 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.151510 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.151969 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.152464 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.153914 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.154327 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.155058 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.156468 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.156852 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.157291 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.158643 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.159008 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.159422 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.201002 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
I20240426 13:18:08.217303 23796 master.cc:537] Master@127.23.61.62:39733 shutting down...
I20240426 13:18:08.329535 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.330340 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.330744 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: stopping tablet replica
I20240426 13:18:08.349329 23796 master.cc:552] Master@127.23.61.62:39733 shutdown complete.
20b204c02fe14d9eb754d5a169d43996  1
cad2f0c769574fa4ba4bd895fda5a10e  4
decbe9ee3d0a47bab14b976ecaf12843  4
20b204c02fe14d9eb754d5a169d43996  3
cad2f0c769574fa4ba4bd895fda5a10e  3
decbe9ee3d0a47bab14b976ecaf12843  3
20b204c02fe14d9eb754d5a169d43996  1
cad2f0c769574fa4ba4bd895fda5a10e  0
decbe9ee3d0a47bab14b976ecaf12843  8
20b204c02fe14d9eb754d5a169d43996  3
cad2f0c769574fa4ba4bd895fda5a10e  3
decbe9ee3d0a47bab14b976ecaf12843  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (30556 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240426 13:18:08.389961 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:42989
I20240426 13:18:08.391049 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.396344 24319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.397265 24320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.398036 24322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.398334 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:08.399277 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.399464 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.399610 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488399593 us; error 0 us; skew 500 ppm
I20240426 13:18:08.400179 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.402446 23796 webserver.cc:460] Webserver started at http://127.23.61.62:46033/ using document root <none> and password file <none>
I20240426 13:18:08.402936 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.403110 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.403393 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.404628 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "947ecbe50c614eaebc7282c4f49c3f4d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.408772 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:18:08.411734 24327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.412432 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240426 13:18:08.412698 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "947ecbe50c614eaebc7282c4f49c3f4d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.412943 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.445961 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:08.447157 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.482138 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:42989
I20240426 13:18:08.482219 24378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:42989 every 8 connection(s)
I20240426 13:18:08.485821 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.494596 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Bootstrap starting.
I20240426 13:18:08.498656 24379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:08.502111 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: No bootstrap required, opened a new log
I20240426 13:18:08.503991 24379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.504456 24379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:08.504645 24379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947ecbe50c614eaebc7282c4f49c3f4d, State: Initialized, Role: FOLLOWER
I20240426 13:18:08.505144 24379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.505559 24379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:18:08.505746 24379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:18:08.505973 24379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:08.509255 24379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.509660 24379 leader_election.cc:302] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947ecbe50c614eaebc7282c4f49c3f4d; no voters: 
I20240426 13:18:08.510591 24379 leader_election.cc:288] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 13:18:08.510933 24382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:08.512231 24382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 LEADER]: Becoming Leader. State: Replica: 947ecbe50c614eaebc7282c4f49c3f4d, State: Running, Role: LEADER
I20240426 13:18:08.512804 24382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.513396 24379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:18:08.518684 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER } }
I20240426 13:18:08.519203 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:08.523356 24384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947ecbe50c614eaebc7282c4f49c3f4d. Latest consensus state: current_term: 1 leader_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER } }
I20240426 13:18:08.523761 24384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:08.525622 24391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:18:08.530272 24391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:18:08.531220 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:18:08.537955 24391 catalog_manager.cc:1347] Generated new cluster ID: e6bf7048ebc44e95b24466848876fdb7
I20240426 13:18:08.538153 24391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:18:08.557147 24391 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:18:08.558137 24391 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:18:08.569741 24391 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Generated new TSK 0
I20240426 13:18:08.570142 24391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:18:08.597218 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.602582 24400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.603715 24401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.605633 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:08.605944 24403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.606760 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.606938 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.607082 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488607065 us; error 0 us; skew 500 ppm
I20240426 13:18:08.607604 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.609886 23796 webserver.cc:460] Webserver started at http://127.23.61.1:45717/ using document root <none> and password file <none>
I20240426 13:18:08.610329 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.610491 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.610718 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.611752 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "615118e943cc40268286df02ea3ee519"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.615875 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240426 13:18:08.618862 24408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.619557 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240426 13:18:08.619814 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "615118e943cc40268286df02ea3ee519"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.620096 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.638633 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:08.639763 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.641170 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.643200 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.643380 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.643582 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.643766 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.680308 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:44527
I20240426 13:18:08.680409 24470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:44527 every 8 connection(s)
I20240426 13:18:08.684490 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.691344 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.692595 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.695143 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.696328 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:08.697463 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.697715 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.697894 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488697876 us; error 0 us; skew 500 ppm
I20240426 13:18:08.698506 24471 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.698513 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.698858 24471 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.699646 24471 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.701520 23796 webserver.cc:460] Webserver started at http://127.23.61.2:46119/ using document root <none> and password file <none>
I20240426 13:18:08.701686 24344 ts_manager.cc:194] Registered new tserver with Master: 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:08.702032 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.702239 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.702474 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.703320 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40754
I20240426 13:18:08.703505 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "9d0a6220c3eb4062b0365da45223700d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.707791 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240426 13:18:08.710731 24483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.711364 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240426 13:18:08.711607 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "9d0a6220c3eb4062b0365da45223700d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.711864 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.739805 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:08.740957 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.742256 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.744328 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.744524 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.744755 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.744902 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.780867 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:33815
I20240426 13:18:08.780954 24545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:33815 every 8 connection(s)
I20240426 13:18:08.785193 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.791112 24549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.793751 24550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.794888 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:08.794832 24552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.796121 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.796319 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.796478 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488796461 us; error 0 us; skew 500 ppm
I20240426 13:18:08.796640 24546 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.796969 24546 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.797053 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.797730 24546 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.799636 24344 ts_manager.cc:194] Registered new tserver with Master: 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:08.799732 23796 webserver.cc:460] Webserver started at http://127.23.61.3:40097/ using document root <none> and password file <none>
I20240426 13:18:08.800412 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.800650 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.800956 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.801388 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40756
I20240426 13:18:08.802350 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.806569 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240426 13:18:08.809448 24557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.810123 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240426 13:18:08.810362 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.810597 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.820912 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:08.821943 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.823215 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.825238 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.825408 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.825623 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.825764 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:08.861052 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:45341
I20240426 13:18:08.861146 24619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:45341 every 8 connection(s)
I20240426 13:18:08.873054 24620 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.873409 24620 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.874050 24620 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.875710 24344 ts_manager.cc:194] Registered new tserver with Master: f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:08.876098 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012216981s
I20240426 13:18:08.877285 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40762
I20240426 13:18:08.879009 23796 test_util.cc:268] Using random seed: -1138823794
I20240426 13:18:08.898370 24344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40772:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20240426 13:18:08.901078 24344 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:18:08.966135 24585 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:08.967533 24585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.969211 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:08.970573 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.973173 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:08.974462 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.977223 24584 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:08.978590 24584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.981333 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:08.981545 24579 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:08.982607 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.984297 24578 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:08.985539 24578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.989166 24580 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:08.990389 24580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.994074 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:08.996295 24579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.996454 24577 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:08.997690 24577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.998836 24511 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:09.000070 24511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.000183 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.003067 24505 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:09.003293 24508 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:09.004324 24505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.005411 24506 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:09.006598 24506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.009110 24507 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:09.010332 24507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.001219 24510 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:09.013177 24509 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:09.014346 24508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.014987 24509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.015859 24510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.017534 24503 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:09.018829 24503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.021173 24504 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:09.022414 24504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.028445 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:09.029744 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.042285 24635 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.074357 24436 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:09.075729 24436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.077368 24435 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:09.078662 24435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.080965 24635 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.083711 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:09.085079 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.089630 24433 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:09.091351 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.097406 24429 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:09.098904 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.106751 24432 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:09.107314 24431 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:09.107506 24430 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:09.112295 24427 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:09.109544 24428 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:09.113883 24635 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.114344 24635 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.072s	user 0.012s	sys 0.013s
I20240426 13:18:09.114921 24432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.115674 24431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.116382 24427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.116959 24430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.117067 24635 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.117578 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.117831 24635 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.118171 24635 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.125504 24636 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.120594 24635 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.161790 24636 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.174407 24638 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.186672 24546 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.187053 24635 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.072s	user 0.002s	sys 0.012s
I20240426 13:18:09.188596 24635 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.195530 24636 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.196161 24636 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.071s	user 0.006s	sys 0.009s
I20240426 13:18:09.199569 24635 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.207674 24638 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.200966 24636 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.208408 24636 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.208778 24636 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.209602 24635 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.209551 24636 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.210325 24635 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.000s
I20240426 13:18:09.215066 24638 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.215582 24638 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.042s	user 0.010s	sys 0.002s
I20240426 13:18:09.217043 24635 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.218106 24635 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.218176 24638 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.219213 24638 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.219573 24638 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.220141 24620 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.218693 24635 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.220779 24636 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.024s	user 0.015s	sys 0.000s
I20240426 13:18:09.220896 24635 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.221792 24636 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.223476 24635 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20240426 13:18:09.224314 24635 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.220314 24638 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.228085 24636 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.229943 24635 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.234757 24638 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.009s
I20240426 13:18:09.235739 24471 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.241519 24635 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.242075 24635 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.018s	user 0.017s	sys 0.000s
I20240426 13:18:09.242100 24638 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.244905 24635 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.245760 24635 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.246107 24635 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.246970 24635 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.250631 24638 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.250941 24635 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20240426 13:18:09.251917 24635 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.257706 24635 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.266108 24636 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.266650 24636 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.045s	user 0.020s	sys 0.020s
I20240426 13:18:09.266675 24638 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.267400 24638 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.018s
I20240426 13:18:09.268990 24635 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.269480 24635 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.001s
I20240426 13:18:09.269610 24636 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.270411 24636 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.270053 24638 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.270749 24636 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.270879 24638 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.271257 24638 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.271553 24636 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.272079 24638 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.272281 24635 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.273150 24635 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.273492 24635 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.273955 24636 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20240426 13:18:09.274544 24638 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.002s
I20240426 13:18:09.274857 24636 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.274363 24635 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.275444 24638 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.278316 24635 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20240426 13:18:09.279309 24635 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.280894 24636 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.281128 24638 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.285032 24635 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.291754 24638 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.292430 24638 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.003s
I20240426 13:18:09.295636 24638 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.301746 24638 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.301959 24636 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.302155 24638 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.302541 24636 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.028s	user 0.016s	sys 0.008s
I20240426 13:18:09.302897 24638 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.305251 24638 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20240426 13:18:09.300983 24635 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.306006 24635 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.007s
I20240426 13:18:09.305613 24636 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.306201 24638 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.306486 24636 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.306912 24636 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.307737 24636 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.308701 24635 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.309562 24635 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.309895 24635 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.310331 24636 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20240426 13:18:09.310716 24635 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.311314 24636 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.312804 24638 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.315220 24635 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.005s
I20240426 13:18:09.316170 24635 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.317414 24636 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.321439 24639 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.322039 24639 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.324429 24638 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.324888 24638 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.003s
I20240426 13:18:09.327409 24638 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.327881 24638 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.328250 24638 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.328359 24636 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.328822 24636 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20240426 13:18:09.328837 24638 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.331459 24636 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.332386 24636 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.332789 24636 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.335631 24638 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20240426 13:18:09.333693 24636 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.336740 24639 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:09.337234 24638 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.338625 24636 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240426 13:18:09.344879 24636 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.346769 24638 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.350418 24635 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.350546 24636 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.352586 24639 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.356182 24639 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.358726 24639 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.361819 24639 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.367041 24638 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.367720 24638 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.031s	user 0.012s	sys 0.008s
I20240426 13:18:09.364281 24639 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.370357 24639 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.372495 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.372627 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.374665 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.375552 24445 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.376683 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.377914 24594 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.380120 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
W20240426 13:18:09.380635 24486 leader_election.cc:341] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.381659 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.382267 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.382968 24593 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.382472 24484 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.384166 24636 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.384579 24636 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.040s	user 0.013s	sys 0.013s
I20240426 13:18:09.384596 24484 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.385551 24639 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.385898 24639 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.386345 24639 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.386374 24657 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.386848 24657 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.387056 24657 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.387087 24636 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.387847 24636 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.385937 24484 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.388234 24636 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
W20240426 13:18:09.389027 24486 leader_election.cc:341] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.389122 24636 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.391433 24636 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20240426 13:18:09.393887 24644 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.394209 24638 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.394759 24636 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.394435 24644 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.394989 24638 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.395328 24638 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.397583 24638 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.400071 24644 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.412480 24657 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.413930 24643 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.414409 24643 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.414870 24635 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.415191 24657 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.415406 24635 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.099s	user 0.009s	sys 0.009s
I20240426 13:18:09.416805 24643 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.417346 24644 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.417841 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.419368 24593 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.415926 24657 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.423754 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.424507 24444 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.427304 24636 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.417935 24644 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.430141 24644 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.430200 24593 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.431504 24484 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.422940 24657 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.435061 24657 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.435993 24444 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.436132 24660 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.436688 24635 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.444698 24639 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.456952 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.457813 24593 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.461592 24639 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:09.464784 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.465267 24638 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.097s	user 0.011s	sys 0.013s
I20240426 13:18:09.466306 24638 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
W20240426 13:18:09.467444 24486 leader_election.cc:341] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.469779 24662 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.470211 24662 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.460403 24657 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.473441 24662 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.474221 24657 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.475350 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.475937 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.478231 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.479305 24638 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.446951 24635 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.485023 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.486410 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.487632 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
W20240426 13:18:09.482452 24486 leader_election.cc:341] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:09.489261 24411 leader_election.cc:341] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.471963 24593 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
W20240426 13:18:09.490135 24411 leader_election.cc:341] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:09.490892 24411 leader_election.cc:341] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.491014 24484 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.491744 24657 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.495961 24595 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.481495 24635 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.497347 24657 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.499261 24484 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.500365 24639 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.500061 24635 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.500775 24639 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.501502 24657 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.504207 24636 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.504689 24636 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.110s	user 0.042s	sys 0.016s
I20240426 13:18:09.501518 24639 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.507195 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.507369 24636 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.507819 24444 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.508129 24636 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.508466 24636 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.509351 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.509204 24636 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.510185 24643 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.510546 24643 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.510834 24643 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.517959 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.518994 24643 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.519449 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.498340 24660 raft_consensus.cc:686] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.520138 24594 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 0.
W20240426 13:18:09.526891 24409 leader_election.cc:341] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.527500 24409 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
W20240426 13:18:09.527760 24392 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:09.528728 24409 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:09.529446 24644 raft_consensus.cc:2726] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.529940 24662 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.530138 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.530303 24662 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.530668 24662 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.530966 24668 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.531430 24668 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.533748 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.534238 24643 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.536170 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.536509 24668 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.534330 24444 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:09.537547 24486 leader_election.cc:341] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.540733 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.541348 24594 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.542356 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.543427 24668 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.543816 24668 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.544224 24668 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.545779 24662 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.548413 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
W20240426 13:18:09.549640 24411 leader_election.cc:341] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.550559 24668 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.553112 24668 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.553318 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.553447 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.553864 24594 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:09.554862 24486 leader_election.cc:341] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.556535 24444 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
W20240426 13:18:09.557618 24409 leader_election.cc:341] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.557569 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.558125 24409 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.558697 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:09.528957 24660 consensus_queue.cc:237] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.559233 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.564224 24643 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.564718 24643 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:09.565521 24643 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.566696 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.567294 24595 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.567476 24636 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.062s	user 0.015s	sys 0.009s
I20240426 13:18:09.568166 24644 raft_consensus.cc:2726] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.568917 24636 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.569051 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.575052 24594 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.569386 24662 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.576339 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.578065 24657 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.578626 24657 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.579484 24657 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.580333 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.582103 24638 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.582594 24638 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.116s	user 0.019s	sys 0.008s
I20240426 13:18:09.586370 24635 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.171s	user 0.016s	sys 0.013s
I20240426 13:18:09.586057 24342 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:09.587291 24635 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.575102 24595 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 1.
I20240426 13:18:09.588832 24409 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:09.591863 24638 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.592996 24638 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.593428 24638 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.594481 24638 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.595731 24662 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.596840 24636 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.600726 24635 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.601372 24662 raft_consensus.cc:686] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:09.604661 24662 consensus_queue.cc:237] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.603606 24638 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.004s
I20240426 13:18:09.608716 24638 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.609788 24635 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.610354 24635 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.002s
I20240426 13:18:09.620340 24635 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.621107 24635 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.621383 24635 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.622522 24638 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.623396 24636 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.623773 24636 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.055s	user 0.011s	sys 0.004s
I20240426 13:18:09.624275 24635 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.626438 24636 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.627380 24636 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.627689 24636 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.628549 24657 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.628988 24635 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.003s
I20240426 13:18:09.628973 24657 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.630085 24635 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.630936 24657 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.631907 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.632299 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.633059 24636 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
W20240426 13:18:09.633695 24486 leader_election.cc:341] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.634002 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.636433 24635 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.637181 24484 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.638262 24657 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.638609 24657 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.638929 24657 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.640372 24344 catalog_manager.cc:5484] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.647361 24340 catalog_manager.cc:5484] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.650734 24636 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20240426 13:18:09.651964 24636 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.659325 24636 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.660248 24657 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.664126 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.664884 24657 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
W20240426 13:18:09.665987 24486 leader_election.cc:341] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.664211 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.666885 24595 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.675321 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.677276 24636 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.677822 24636 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.010s
I20240426 13:18:09.678397 24484 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.679196 24657 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.679782 24657 raft_consensus.cc:686] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.680809 24657 consensus_queue.cc:237] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.682322 24638 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.682739 24638 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.074s	user 0.037s	sys 0.024s
I20240426 13:18:09.685236 24638 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.685700 24638 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.685957 24638 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.686436 24635 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.686604 24638 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.686911 24636 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.688931 24638 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20240426 13:18:09.691289 24635 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.061s	user 0.017s	sys 0.038s
I20240426 13:18:09.691797 24638 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.693717 24635 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.694471 24635 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.695099 24635 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.698490 24638 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.695756 24635 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.689527 24636 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.698900 24636 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.699540 24636 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.701515 24636 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20240426 13:18:09.699987 24635 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20240426 13:18:09.702462 24635 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.708557 24636 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.711007 24635 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.711658 24638 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.712155 24638 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.007s
I20240426 13:18:09.714161 24636 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.715170 24638 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.715828 24638 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.716085 24638 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.716648 24638 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.718592 24638 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.003s
I20240426 13:18:09.720057 24638 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.719641 24344 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.721868 24344 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.725454 24638 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.723587 24344 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.728080 24344 catalog_manager.cc:5484] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.731983 24636 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.732611 24636 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.008s
I20240426 13:18:09.734892 24635 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.735280 24638 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.735285 24635 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.013s
I20240426 13:18:09.735227 24636 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.735857 24638 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.004s
I20240426 13:18:09.736019 24636 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.736398 24636 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.737728 24635 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.738166 24635 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.738523 24635 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.738165 24638 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.737090 24636 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.738902 24638 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.739209 24638 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.739053 24635 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.739917 24638 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.740447 24636 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:18:09.741080 24635 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I20240426 13:18:09.741475 24636 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.741859 24635 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.745064 24638 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.002s
I20240426 13:18:09.745828 24638 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.747097 24635 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.750900 24636 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.756224 24638 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.758471 24635 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.758881 24635 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.012s
I20240426 13:18:09.761305 24635 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.761966 24635 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.762213 24635 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.762859 24635 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.763448 24636 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.764169 24636 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.000s
I20240426 13:18:09.765513 24635 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:09.767700 24636 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.768383 24636 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.768657 24636 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.769250 24636 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.771152 24636 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240426 13:18:09.771831 24638 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.772276 24638 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.003s
I20240426 13:18:09.774896 24638 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.775534 24638 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.775797 24638 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.776346 24638 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.777889 24638 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240426 13:18:09.785792 24657 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.786157 24657 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.787734 24657 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.788341 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.788650 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.788952 24444 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.789222 24595 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.789901 24486 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:09.790591 24657 raft_consensus.cc:2781] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.790895 24657 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.791152 24657 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.795398 24657 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.797232 24657 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.797657 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.797773 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.798100 24444 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.798226 24595 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.802213 24444 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.802593 24595 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.803050 24486 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:09.803748 24657 raft_consensus.cc:2781] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.804137 24657 raft_consensus.cc:686] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.804713 24657 consensus_queue.cc:237] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.811292 24343 catalog_manager.cc:5484] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.842130 24643 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.842592 24643 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.844118 24643 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.844971 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.845068 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.845464 24444 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.845587 24520 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.846278 24560 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.846877 24643 raft_consensus.cc:2781] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.847141 24643 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.847384 24643 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.851436 24643 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.852758 24643 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.853446 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.853628 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:09.853878 24444 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.854102 24520 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.857920 24444 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.857961 24520 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.858786 24560 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:09.859344 24643 raft_consensus.cc:2781] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.859699 24643 raft_consensus.cc:686] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:09.860435 24643 consensus_queue.cc:237] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.866593 24343 catalog_manager.cc:5484] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:09.880455 24662 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.880861 24662 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.882242 24662 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.883097 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.883008 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.883901 24595 raft_consensus.cc:2379] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 615118e943cc40268286df02ea3ee519 in current term 1: Already voted for candidate 9d0a6220c3eb4062b0365da45223700d in this term.
I20240426 13:18:09.884760 24409 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.885308 24662 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.889380 24662 raft_consensus.cc:2726] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.924078 24657 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.924553 24657 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.926090 24657 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.926822 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.927055 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.927474 24444 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.927588 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.928504 24486 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:09.929098 24657 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.929351 24657 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.929591 24657 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.933810 24657 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.935252 24657 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.935971 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.936146 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.936477 24444 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.936616 24595 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.940594 24444 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.941002 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.941525 24486 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:09.942277 24657 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.942631 24657 raft_consensus.cc:686] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.943362 24657 consensus_queue.cc:237] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.949604 24343 catalog_manager.cc:5484] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.959749 24662 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.960172 24662 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.961649 24662 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.962337 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.962605 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.962944 24595 raft_consensus.cc:2379] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 615118e943cc40268286df02ea3ee519 in current term 1: Already voted for candidate 9d0a6220c3eb4062b0365da45223700d in this term.
I20240426 13:18:09.963798 24409 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.964414 24662 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.968488 24662 raft_consensus.cc:2726] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.994741 24657 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.995182 24657 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.996703 24657 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.997684 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.997897 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.998284 24595 raft_consensus.cc:2379] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d0a6220c3eb4062b0365da45223700d in current term 1: Already voted for candidate 615118e943cc40268286df02ea3ee519 in this term.
I20240426 13:18:09.999174 24484 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d; no voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.999892 24657 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.001950 24660 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.005141 24657 raft_consensus.cc:2726] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:10.013494 24657 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.031818 24657 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.040792 24670 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.062981 24444 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.064364 24660 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.070236 24660 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.074573 24670 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.082999 24446 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.088748 24668 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.098644 24668 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.109489 24660 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.115645 24643 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.140651 24643 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.145081 24519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.146026 24519 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.165841 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:10.166289 24595 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.170367 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.170780 24520 raft_consensus.cc:595] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.174588 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.175026 24520 raft_consensus.cc:595] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.188056 24662 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.198786 24718 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.320930 24714 consensus_queue.cc:1035] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.330791 24714 consensus_queue.cc:1035] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.331158 24660 consensus_queue.cc:1035] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.343453 24668 consensus_queue.cc:1035] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.362241 24668 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.371907 24657 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.379263 24657 raft_consensus.cc:979] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.379751 24657 raft_consensus.cc:1067] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.381240 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.381721 24445 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.381994 24445 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.386639 24445 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.388231 24445 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.388914 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.389230 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.389523 24520 raft_consensus.cc:3032] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.389717 24595 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.389906 24520 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.390697 24520 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.391970 24520 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.394752 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.395709 24409 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:10.396462 24718 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.396890 24718 raft_consensus.cc:686] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.397209 24520 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.397612 24718 consensus_queue.cc:237] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.403208 24342 catalog_manager.cc:5484] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.559386 24668 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.559813 24657 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.561215 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.561741 24445 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.562111 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.566792 24445 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.568365 24445 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:10.568876 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.569278 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.569644 24595 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.569753 24520 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.570057 24520 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.570634 24520 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.571516 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.575022 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.575611 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.575986 24409 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:10.576733 24718 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.577109 24718 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.577716 24718 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.584347 24342 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:10.629823 24657 raft_consensus.cc:979] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.630251 24657 raft_consensus.cc:1067] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.631603 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.632161 24445 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.632544 24445 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.636821 24445 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.638070 24445 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.638852 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.638979 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.639343 24520 raft_consensus.cc:3032] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.639461 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.639640 24520 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.640322 24520 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.641438 24520 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.645851 24643 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.646262 24714 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.646535 24520 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.646960 24595 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.647437 24411 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:10.647687 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:10.648239 24718 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.648365 24445 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.648648 24718 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.648770 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.649426 24718 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.653869 24445 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.655990 24445 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.655438 24342 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.656229 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.656934 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.656833 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.657418 24595 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.657718 24595 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:10.658403 24595 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.659564 24595 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.663019 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.663971 24411 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:10.664644 24718 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.665110 24718 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.666043 24595 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.665992 24718 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.671600 24342 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.927685 24520 raft_consensus.cc:1261] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:10.928807 24718 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:10.937393 24595 raft_consensus.cc:1261] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:10.938670 24718 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.071800 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.073516 24719 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.074016 24520 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.073958 24595 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.075450 24719 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.075865 24718 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.087199 24519 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.095615 24595 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.099464 24595 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.100730 24719 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:18:11.104676 24732 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:11.105891 24718 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:13.208653 23796 auto_leader_rebalancer-test.cc:331] The leader distribution is 
I20240426 13:18:13.209298 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:13.224800 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:13.225255 24445 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:13.227192 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:13.227594 24445 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:13.228441 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 13:18:13.228821 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:13.582510 24719 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.582907 24719 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.584195 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:13.584620 24594 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:13.584878 24594 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.589005 24594 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.590376 24594 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:13.591198 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:13.591296 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:13.591856 24519 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.591885 24445 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:13.592396 24445 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:13.593106 24445 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.594417 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.598174 24519 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.599573 24560 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:13.600337 24672 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:13.600582 24445 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.601294 24672 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:13.602145 24672 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.609355 24343 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:13.635968 24739 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.636442 24719 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.637728 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:13.638198 24594 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:13.638473 24594 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.642700 24594 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.644145 24594 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:13.645009 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:13.645126 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:13.645649 24445 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:13.645704 24519 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.646097 24445 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:13.646862 24445 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.648146 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.652216 24519 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.653234 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:13.653889 24672 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:13.654012 24445 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.654302 24672 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:13.655169 24672 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.661741 24343 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:13.997252 24445 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:13.998425 24672 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.006915 24519 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.008442 24672 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.153931 24519 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.154987 24672 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.162557 24445 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.164521 24740 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.229691 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:14.234771 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:14.235152 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:15.235841 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:15.241791 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:15.242154 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:16.242888 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:16.247956 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:16.248363 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:17.249097 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:17.255771 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:17.256155 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:18.257117 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:18.262763 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:18.263126 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:19.263839 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:19.269687 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:19.270059 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:20.270879 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:20.275799 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:20.276223 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:21.277122 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:21.282230 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:21.282624 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:22.283318 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:22.289212 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:22.289563 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:23.311794 23796 auto_leader_rebalancer-test.cc:350] The leader distribution is 
I20240426 13:18:23.316251 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:23.316884 24595 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.322889 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:23.323289 24595 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.331246 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.331801 24520 raft_consensus.cc:595] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.335915 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.336381 24520 raft_consensus.cc:595] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.340257 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.340785 24520 raft_consensus.cc:595] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.343062 24761 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.343607 24764 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.345183 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:23.345618 24445 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.345944 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.351711 24445 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.353560 24445 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:23.354269 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.354519 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.355145 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.355159 24595 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:18:23.355583 24595 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:23.356215 24595 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.357115 24595 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.359797 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.360937 24411 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:23.361765 24762 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:18:23.362149 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.362782 24762 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.363649 24762 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.369774 24343 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 3 to 4, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 4 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:23.386794 24757 raft_consensus.cc:979] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.387153 24757 raft_consensus.cc:1067] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.388252 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.388655 24445 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.388911 24445 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.393175 24445 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.394405 24445 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.395146 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.395236 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.395684 24520 raft_consensus.cc:3032] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.395721 24595 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.396095 24520 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.396718 24520 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.397747 24520 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.399739 24595 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.400589 24409 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:23.401198 24762 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.401516 24762 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.402290 24762 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.402563 24520 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.408191 24343 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.524778 24771 raft_consensus.cc:979] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.525189 24757 raft_consensus.cc:1067] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.526356 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.526706 24445 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.526902 24445 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.531055 24445 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.532727 24445 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.532756 24761 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.533084 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.533383 24761 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.533653 24520 raft_consensus.cc:3032] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.533522 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.534044 24520 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.534237 24595 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.534785 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:23.534790 24520 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.535408 24445 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.535760 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.535916 24520 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.539994 24595 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.540161 24520 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.540997 24409 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:23.541409 24445 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.541615 24762 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.542105 24762 raft_consensus.cc:686] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.542909 24445 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.542865 24762 consensus_queue.cc:237] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.543870 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.544075 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.544415 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.544525 24595 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:18:23.548221 24595 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:23.548969 24595 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.550002 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.550155 24595 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.550536 24343 catalog_manager.cc:5484] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.552769 24411 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters: 
I20240426 13:18:23.553547 24762 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:18:23.554190 24762 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.555305 24762 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.555636 24595 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.561604 24343 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 3 to 4, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 4 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.583031 24771 raft_consensus.cc:979] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.583482 24757 raft_consensus.cc:1067] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.584676 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6"
dest_uuid: "615118e943cc40268286df02ea3ee519"
 from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.585129 24445 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.585480 24445 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.589217 24445 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.590251 24445 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.590955 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.591090 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.591383 24520 raft_consensus.cc:3032] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.591523 24595 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.591645 24520 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.592348 24520 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.593183 24520 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.595335 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.596082 24409 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:23.596669 24762 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.596738 24520 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.597034 24762 raft_consensus.cc:686] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.597955 24762 consensus_queue.cc:237] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.604017 24343 catalog_manager.cc:5484] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.791553 24595 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.792573 24762 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.802284 24520 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.804806 24779 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.815157 24595 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.816138 24779 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:23.823470 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.825613 24762 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:23.961488 24595 raft_consensus.cc:1261] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.962568 24762 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.970515 24520 raft_consensus.cc:1261] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.971843 24762 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.994836 24520 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.995630 24762 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:24.005477 24595 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:24.006863 24787 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:24.009928 24520 raft_consensus.cc:1261] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:24.011371 24787 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:24.022145 24594 raft_consensus.cc:1261] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:24.024140 24762 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:26.368892 23796 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20240426 13:18:26.369436 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:26.376086 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "9d0a6220c3eb4062b0365da45223700d"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.376513 24445 raft_consensus.cc:595] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Received request to transfer leadership to TS 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:26.378425 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.378773 24445 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:26.380465 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.380836 24445 raft_consensus.cc:595] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:26.382557 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "9d0a6220c3eb4062b0365da45223700d"
 from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.382889 24445 raft_consensus.cc:595] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:26.383544 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:18:26.383893 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:26.560498 24793 raft_consensus.cc:979] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: : Instructing follower 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.560904 24787 raft_consensus.cc:1067] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.562018 24520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
 from {username='slave'} at 127.0.0.1:41888
I20240426 13:18:26.562418 24520 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.562741 24520 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.565198 24793 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.565570 24787 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.566702 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:26.567184 24594 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.567495 24594 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.567891 24520 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.570231 24520 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:26.570966 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:26.570966 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.571664 24595 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.571673 24445 raft_consensus.cc:3032] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:26.572223 24445 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.572909 24445 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.573076 24594 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.574041 24445 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.575239 24594 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 5 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:26.576020 24595 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 3.
I20240426 13:18:26.576788 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.577287 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:26.577323 24446 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:18:26.577245 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:26.577937 24446 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.578397 24757 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:26.578686 24446 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.579495 24445 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 3.
I20240426 13:18:26.580284 24446 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.580570 24757 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:26.581323 24757 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.577934 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.586889 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 5.
I20240426 13:18:26.587241 24446 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 5.
I20240426 13:18:26.587873 24560 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:26.588810 24792 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:18:26.590252 24792 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:26.590982 24792 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.591233 24343 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 3 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.598729 24342 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 4 to 5, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.703052 24793 raft_consensus.cc:979] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: : Instructing follower 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.703416 24787 raft_consensus.cc:1067] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Signalling peer 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.704604 24520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
 from {username='slave'} at 127.0.0.1:41888
I20240426 13:18:26.705041 24520 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.705307 24520 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.709611 24520 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.710845 24520 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 2 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:26.711594 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.711720 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:26.712291 24446 raft_consensus.cc:3032] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:26.712342 24595 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.712574 24446 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.713322 24446 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.714378 24446 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.716688 24595 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 2.
I20240426 13:18:26.717475 24484 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:26.718081 24757 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:26.718400 24757 raft_consensus.cc:686] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:26.718822 24446 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 2.
I20240426 13:18:26.718963 24757 consensus_queue.cc:237] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.725360 24342 catalog_manager.cc:5484] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 1 to 2, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 2 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.771771 24793 raft_consensus.cc:979] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.772209 24787 raft_consensus.cc:1067] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.773352 24595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
 from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:26.773756 24595 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.774019 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.777953 24595 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.779239 24595 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:26.779961 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.780230 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:26.780460 24446 raft_consensus.cc:3032] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:26.780681 24520 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.780776 24446 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.781524 24446 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.782426 24446 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.784751 24520 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:26.785586 24560 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 
I20240426 13:18:26.786203 24792 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:26.786659 24792 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:26.786937 24446 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:26.787550 24792 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.793169 24342 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:26.984022 24595 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:26.985570 24757 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:26.994307 24446 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:26.996204 24757 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.046972 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:18:27.048007 24792 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:18:27.055377 24446 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:18:27.056818 24810 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:18:27.148497 24446 raft_consensus.cc:1261] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:27.149484 24757 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:27.156728 24595 raft_consensus.cc:1261] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:27.158149 24813 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:27.292424 24520 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:27.293480 24810 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.301693 24446 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:27.303467 24810 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.384740 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:27.390419 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:27.390812 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:28.391917 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:28.399250 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:28.399643 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:29.400456 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:29.407269 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:29.407652 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:30.408588 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:30.414304 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:30.414804 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:31.415517 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:31.420588 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:31.421066 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:32.421937 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:32.427043 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:32.427412 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:33.428278 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:33.433764 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:33.434232 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:34.435015 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:34.440019 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:34.440415 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:35.441082 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:35.445895 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:35.446224 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:36.467764 23796 auto_leader_rebalancer-test.cc:376] The leader distribution is 
I20240426 13:18:36.468273 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
W20240426 13:18:36.480716 24560 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20240426 13:18:36.484844 24560 consensus_peers.cc:485] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:36.495002 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.495594 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.496347 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.496842 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.498685 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.499121 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.499514 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.501117 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.501559 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.502264 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.503873 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.504366 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.504737 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.506340 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.506794 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.507529 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.509121 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.509547 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.509894 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.511482 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.511910 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.512351 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.514443 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.514966 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.515626 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.517555 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.518245 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.519232 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.521179 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.521652 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.522013 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.550900 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
I20240426 13:18:36.568535 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
W20240426 13:18:36.583885 24486 consensus_peers.cc:485] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:36.586889 24560 consensus_peers.cc:485] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:36.601452 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.602129 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.603081 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.604069 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.605715 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.606124 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.606454 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.607894 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.608330 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.608952 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.610339 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.610819 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.611155 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.612932 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.613380 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.613842 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.615521 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.615931 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.616333 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.617798 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.618211 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.618566 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.620173 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.620620 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.621031 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.622705 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.623172 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.623528 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.624998 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.625375 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.626062 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.652815 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:18:36.669242 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
I20240426 13:18:36.691056 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.691653 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.692263 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.692701 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.694380 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.694859 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.695268 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.696763 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.697222 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.697584 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.698976 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.699406 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.699774 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.701236 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.701679 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.702355 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.703748 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.704244 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.704933 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.706432 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.706889 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.707235 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.708731 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.709126 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.709465 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.710916 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.711352 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.712158 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.713629 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.714089 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.714416 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.741904 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
I20240426 13:18:36.758637 23796 master.cc:537] Master@127.23.61.62:42989 shutting down...
I20240426 13:18:36.783674 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.784241 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.784546 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: stopping tablet replica
I20240426 13:18:36.802116 23796 master.cc:552] Master@127.23.61.62:42989 shutdown complete.
615118e943cc40268286df02ea3ee519  5
9d0a6220c3eb4062b0365da45223700d  4
f6a9d1d3ef9d41489d7ab9a93c740798  1
615118e943cc40268286df02ea3ee519  3
9d0a6220c3eb4062b0365da45223700d  4
f6a9d1d3ef9d41489d7ab9a93c740798  3
615118e943cc40268286df02ea3ee519  8
9d0a6220c3eb4062b0365da45223700d  1
f6a9d1d3ef9d41489d7ab9a93c740798  1
615118e943cc40268286df02ea3ee519  4
9d0a6220c3eb4062b0365da45223700d  3
f6a9d1d3ef9d41489d7ab9a93c740798  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28452 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20240426 13:18:36.842198 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:34153
I20240426 13:18:36.843310 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:36.848953 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:36.849054 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:36.849535 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:36.851233 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:36.852423 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:36.852622 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:36.852799 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137516852781 us; error 0 us; skew 500 ppm
I20240426 13:18:36.853384 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:36.860855 23796 webserver.cc:460] Webserver started at http://127.23.61.62:34839/ using document root <none> and password file <none>
I20240426 13:18:36.861356 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:36.861528 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:36.861795 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:36.862982 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "e2ab89ff7e18462eb10e19320ff8beaa"
format_stamp: "Formatted at 2024-04-26 13:18:36 on dist-test-slave-jx77"
I20240426 13:18:36.867147 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240426 13:18:36.870378 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:36.871083 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240426 13:18:36.871344 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "e2ab89ff7e18462eb10e19320ff8beaa"
format_stamp: "Formatted at 2024-04-26 13:18:36 on dist-test-slave-jx77"
I20240426 13:18:36.871628 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:36.896713 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:36.897953 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:36.933480 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:34153
I20240426 13:18:36.933584 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:34153 every 8 connection(s)
I20240426 13:18:36.937157 24889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:36.946286 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Bootstrap starting.
I20240426 13:18:36.950181 24889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:36.953986 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: No bootstrap required, opened a new log
I20240426 13:18:36.955850 24889 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.956326 24889 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:36.956614 24889 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ab89ff7e18462eb10e19320ff8beaa, State: Initialized, Role: FOLLOWER
I20240426 13:18:36.957108 24889 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.957564 24889 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:18:36.957779 24889 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:18:36.958047 24889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:36.961709 24889 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.962178 24889 leader_election.cc:302] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ab89ff7e18462eb10e19320ff8beaa; no voters: 
I20240426 13:18:36.963203 24889 leader_election.cc:288] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 13:18:36.963474 24892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:36.964784 24892 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 LEADER]: Becoming Leader. State: Replica: e2ab89ff7e18462eb10e19320ff8beaa, State: Running, Role: LEADER
I20240426 13:18:36.965428 24892 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.965955 24889 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:18:36.970970 24894 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ab89ff7e18462eb10e19320ff8beaa. Latest consensus state: current_term: 1 leader_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER } }
I20240426 13:18:36.971554 24894 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:36.972329 24893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER } }
I20240426 13:18:36.972930 24893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:36.974047 24902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:18:36.981303 24902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:18:36.982208 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:18:36.989519 24902 catalog_manager.cc:1347] Generated new cluster ID: 7722385be3804d228a41d0506276be2b
I20240426 13:18:36.989749 24902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:18:37.010066 24902 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:18:37.011257 24902 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:18:37.022332 24902 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Generated new TSK 0
I20240426 13:18:37.022873 24902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:18:37.048333 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.053784 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.054636 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.055977 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.056763 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:37.057502 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.057677 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.057793 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517057783 us; error 0 us; skew 500 ppm
I20240426 13:18:37.058274 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.060441 23796 webserver.cc:460] Webserver started at http://127.23.61.1:35125/ using document root <none> and password file <none>
I20240426 13:18:37.060858 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.061013 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.061266 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.062302 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.066541 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:18:37.069633 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.070298 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240426 13:18:37.070556 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.070799 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.094377 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:37.095510 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.096913 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.099117 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.099287 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.099519 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.099678 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.140969 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:41817
I20240426 13:18:37.141070 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:41817 every 8 connection(s)
I20240426 13:18:37.145283 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.152326 24985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.153764 24986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.156459 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:37.157829 24988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.158686 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.158910 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.159071 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517159056 us; error 0 us; skew 500 ppm
I20240426 13:18:37.159791 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.159787 24981 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.160257 24981 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.160998 24981 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.162374 23796 webserver.cc:460] Webserver started at http://127.23.61.2:39467/ using document root <none> and password file <none>
I20240426 13:18:37.163033 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.163290 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.163198 24854 ts_manager.cc:194] Registered new tserver with Master: 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:37.163679 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.165220 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "46106ff992984d33bb4a4d8f30162639"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.165660 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39836
I20240426 13:18:37.169842 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20240426 13:18:37.173144 24993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.173852 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20240426 13:18:37.174106 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "46106ff992984d33bb4a4d8f30162639"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.174358 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.195663 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:37.196897 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.198227 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.200433 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.200621 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.200835 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.201010 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.237288 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:43671
I20240426 13:18:37.237373 25055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:43671 every 8 connection(s)
I20240426 13:18:37.241653 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.247750 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.249310 25060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.250761 25062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.251791 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:37.252781 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.253001 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.253172 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517253146 us; error 0 us; skew 500 ppm
I20240426 13:18:37.253758 25056 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.253752 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.254179 25056 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.254832 25056 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.256438 23796 webserver.cc:460] Webserver started at http://127.23.61.3:45639/ using document root <none> and password file <none>
I20240426 13:18:37.256829 24854 ts_manager.cc:194] Registered new tserver with Master: 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:37.256907 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.257206 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.257457 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.258100 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39848
I20240426 13:18:37.258579 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.262818 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240426 13:18:37.265904 25067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.266606 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20240426 13:18:37.266903 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.267154 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.288144 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:37.289194 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.290521 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.292704 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.292876 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.293118 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.293255 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:37.329595 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:38089
I20240426 13:18:37.329687 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:38089 every 8 connection(s)
I20240426 13:18:37.341786 25130 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.342133 25130 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.342778 25130 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.344436 24854 ts_manager.cc:194] Registered new tserver with Master: 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:37.344772 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011837281s
I20240426 13:18:37.345616 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39858
I20240426 13:18:37.347024 23796 test_util.cc:268] Using random seed: -1110355777
I20240426 13:18:37.367458 24854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39872:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20240426 13:18:37.373580 24854 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:18:37.577472 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.579034 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.579386 24944 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.580775 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.581259 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.582528 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.585220 24943 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.586447 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.589200 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.590479 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.590533 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.591826 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.593201 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.594519 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.595153 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.596433 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.597169 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.598407 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.601140 24939 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.602380 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.605196 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.606555 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.609135 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.610384 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.613425 25093 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.615087 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.617429 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.618050 25145 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:37.618759 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.621213 25088 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.622548 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.625223 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.625478 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.626467 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.626544 25145 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.627086 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.628460 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.629717 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.634732 25095 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.641392 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.641959 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.642685 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.637342 25091 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.646632 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.639305 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.648916 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.648602 25094 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.651011 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.661221 25095 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.662990 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.671178 25094 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.673141 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.684391 25014 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.685154 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.685724 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.686363 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.687680 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.688956 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.692350 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.693193 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.693643 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.694416 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.697156 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.698418 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.701140 25019 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.702380 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.697155 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.705394 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.705107 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.713137 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.725768 25094 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:37.729112 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.734915 24942 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.751345 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.751631 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.752288 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.736433 25092 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.740459 24940 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.719669 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.755167 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.755297 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.746276 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.756772 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.757977 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.748661 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.750306 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.764757 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.768314 24944 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.769526 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.775228 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.804873 24941 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.805512 24946 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:37.806028 25145 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:37.806178 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.806805 25145 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.189s	user 0.019s	sys 0.020s
I20240426 13:18:37.808923 25016 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.809787 24943 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.811029 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.813645 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.815013 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.815280 25148 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:37.815414 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.810181 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.822288 25091 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.823637 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.830816 25012 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.832160 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.853003 25093 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.855770 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.857807 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:37.853677 25145 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:37.858708 25145 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:37.859046 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.867372 25020 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.868748 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.869307 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.870513 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.875072 25087 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:37.876885 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.876590 25088 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.877725 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.878090 25148 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.878361 25147 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:37.880193 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:18:37.881896 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.880295 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.892767 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.897047 25017 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.898214 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.888609 25015 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.902819 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.859057 25145 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:37.904639 25145 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:37.908771 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.909963 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.915614 25147 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.917304 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.918517 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.918411 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:37.923496 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:37.926959 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.906509 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.928849 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.930061 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.931424 24938 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.932668 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.933267 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:37.934392 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.936722 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.938972 24946 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.940659 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.943996 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.942648 24943 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:37.961514 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.945870 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.936595 25016 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.970690 25130 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:37.973150 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:37.957049 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.002503 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:18:37.978174 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:38.004139 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.982647 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:37.983345 25020 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.986541 25093 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.007699 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.008579 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.987274 25091 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:37.990264 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:37.953018 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:38.017274 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.991297 25086 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:37.983907 25145 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.177s	user 0.003s	sys 0.009s
I20240426 13:18:38.018601 25145 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:37.995186 25149 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.019124 25149 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.019580 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.024718 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.998539 25148 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:37.891194 25021 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:38.004623 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.981627 24944 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:18:38.029587 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.030187 25148 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.215s	user 0.005s	sys 0.014s
I20240426 13:18:38.030505 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.034312 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.040359 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.044459 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:18:38.044231 24840 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54776.8us
I20240426 13:18:38.050207 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:38.051501 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.052965 25149 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.057695 25145 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.068341 25015 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:38.069777 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.073909 25147 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.051357 25148 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.076084 25148 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.076387 25148 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.077122 25148 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:38.083885 25131 tablet.cc:2369] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 13:18:38.085844 25088 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.087085 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.090071 25147 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.212s	user 0.005s	sys 0.009s
I20240426 13:18:38.090942 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:38.100340 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.101540 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
W20240426 13:18:38.102592 25069 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.103657 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.109357 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.112370 24941 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.113957 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.117508 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:38.118702 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.127431 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.128734 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.133718 25147 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.144456 25147 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.144779 25147 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.145462 25147 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.148523 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:38.150168 25087 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.149289 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.158952 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.164444 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.164609 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.168125 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.168211 24981 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:38.181056 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:38.182287 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.189365 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.191462 25145 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.191316 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.191548 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:38.192519 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.192772 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.193048 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:38.194216 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.195780 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:38.197000 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.197291 25013 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.198519 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.200145 25145 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.174s	user 0.013s	sys 0.000s
I20240426 13:18:38.201457 25056 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:38.159952 25093 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.204003 25145 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.206416 25091 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.228255 25145 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.207374 25148 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.177s	user 0.009s	sys 0.012s
I20240426 13:18:38.222730 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.228703 25145 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.229168 25147 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.139s	user 0.003s	sys 0.019s
I20240426 13:18:38.229351 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217453 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.213820 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.229674 25145 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.230813 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.232126 25145 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s	user 0.003s	sys 0.004s
I20240426 13:18:38.233109 25145 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.236304 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217965 25086 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.224579 25090 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.237174 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.237821 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217831 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.239202 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.241080 25145 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.225517 25088 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.244215 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.215530 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.245808 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.246529 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.229369 25148 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.245366 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.256443 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.245357 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.213295 25019 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:38.264731 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.268968 25147 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:38.279460 25148 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.286968 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.287640 25090 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.288264 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.281742 25145 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.285348 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.289825 25145 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.057s	user 0.012s	sys 0.031s
I20240426 13:18:38.290482 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.292508 25145 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.293277 25147 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.293227 25145 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.286551 25093 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.300252 25145 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.300889 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.300957 25145 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.301950 25091 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:18:38.303153 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.303220 25145 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.000s
I20240426 13:18:38.303400 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.304415 25145 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.332094 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.335355 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.163596 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:38.350076 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.179656 24944 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:38.357049 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.376466 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.377954 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.379899 25019 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.381626 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.383262 25015 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.385766 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.384589 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.355283 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.392932 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.395599 24943 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.393014 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.408063 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.409291 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.412300 25019 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.414443 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.417305 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.418910 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.419075 25020 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.421002 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:38.426424 25070 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.427179 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.421013 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.433051 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.433512 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.435139 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.435894 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.441484 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.295799 25092 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.461340 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.466588 25013 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.468446 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.473865 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.480684 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.467995 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.495630 25148 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.496421 25148 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.267s	user 0.012s	sys 0.008s
I20240426 13:18:38.497617 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.499004 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.502832 25149 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.502971 25155 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.513041 25155 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.517927 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.519392 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.520320 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.523912 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.528825 25155 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.532027 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.536592 25157 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.539347 25157 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.543363 24942 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.499835 25148 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.563165 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.563630 25149 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.564766 25149 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.566833 25149 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.567756 25157 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.568092 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.568682 24956 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:38.570217 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.572024 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:18:38.572959 25070 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.573477 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.573762 25149 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:38.579622 25145 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.581264 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.587352 25012 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.570097 25017 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.556891 25148 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.588941 25149 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:38.570058 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.596726 25149 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.597409 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.604624 24945 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.605875 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.610054 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.611248 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.612457 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.613544 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.619005 25147 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.619611 25147 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.351s	user 0.022s	sys 0.008s
I20240426 13:18:38.620436 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.620509 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.620625 25149 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.621606 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.623450 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:38.624037 24956 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.624316 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.593691 25148 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.635706 24956 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:38.638993 25093 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.649586 25013 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.657608 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.654426 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.663815 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.662847 24943 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.669503 25149 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.677589 25166 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.667884 24938 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.678628 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:38.679461 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.669657 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.683959 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.663935 25089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 47) took 1022 ms. Trace:
I20240426 13:18:38.684181 25148 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.686050 25092 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 52) took 1031 ms. Trace:
I20240426 13:18:38.688015 25090 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 51) took 1036 ms. Trace:
I20240426 13:18:38.688159 25088 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 45) took 1014 ms. Trace:
I20240426 13:18:38.696297 25088 rpcz_store.cc:276] 0426 13:18:37.640996 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.224681 (+583685us) service_pool.cc:225] Handling call
0426 13:18:38.244300 (+ 19619us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.245415 (+  1115us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.271742 (+ 26327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.616302 (+344560us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.655355 (+ 39053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.689150 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.702423 25147 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.703464 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.703419 25147 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.703953 25147 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.705761 25090 rpcz_store.cc:276] 0426 13:18:37.651962 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.286829 (+634867us) service_pool.cc:225] Handling call
0426 13:18:38.288445 (+  1616us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.288542 (+    97us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.587085 (+298543us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.687524 (+100439us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.687964 (+   440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.694027 25166 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.699427 25148 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.200s	user 0.007s	sys 0.000s
W20240426 13:18:38.701098 24840 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68490.9us
I20240426 13:18:38.670660 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:38.715652 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.708540 25149 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:38.724191 25019 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20240426 13:18:38.730082 25070 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.731668 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.732003 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.732288 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.733325 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.733606 25091 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 53) took 1076 ms. Trace:
W20240426 13:18:38.733882 24996 leader_election.cc:341] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.738577 25148 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.694818 25092 rpcz_store.cc:276] 0426 13:18:37.654448 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.294941 (+640493us) service_pool.cc:225] Handling call
0426 13:18:38.460631 (+165690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.460754 (+   123us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.572134 (+111380us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.685629 (+113495us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.686000 (+   371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.693457 25089 rpcz_store.cc:276] 0426 13:18:37.641629 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.431506 (+789877us) service_pool.cc:225] Handling call
0426 13:18:38.434172 (+  2666us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.434340 (+   168us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.628846 (+194506us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.663472 (+ 34626us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.663865 (+   393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.745680 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.748142 25166 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.748077 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:38.749526 25070 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:38.748081 24995 leader_election.cc:341] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.750317 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:38.751406 25155 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.752310 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.723881 25147 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.753031 25157 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:38.753551 25069 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.753438 25157 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.753968 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.667615 24946 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.739169 25087 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 56) took 1074 ms. Trace:
I20240426 13:18:38.739670 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.759843 25157 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.770581 25093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 58) took 1099 ms. Trace:
I20240426 13:18:38.770362 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.788829 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.761605 25091 rpcz_store.cc:276] 0426 13:18:37.656907 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.301103 (+644196us) service_pool.cc:225] Handling call
0426 13:18:38.302667 (+  1564us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.302781 (+   114us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.583905 (+281124us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.732922 (+149017us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.733383 (+   461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.778270 25093 rpcz_store.cc:276] 0426 13:18:37.671196 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.638075 (+966879us) service_pool.cc:225] Handling call
0426 13:18:38.657133 (+ 19058us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.657246 (+   113us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.697004 (+ 39758us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.770172 (+ 73168us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.770527 (+   355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.763131 25166 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.762547 25086 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 55) took 1100 ms. Trace:
I20240426 13:18:38.794020 25149 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:38.797530 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.798414 25148 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.798441 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.801273 24939 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 27) took 1082 ms. Trace:
I20240426 13:18:38.802131 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.802588 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 32) took 1010 ms. Trace:
I20240426 13:18:38.802444 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.804675 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.809494 25104 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:38.807394 25086 rpcz_store.cc:276] 0426 13:18:37.662332 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.496395 (+834063us) service_pool.cc:225] Handling call
0426 13:18:38.498500 (+  2105us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.498607 (+   107us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.642656 (+144049us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.739661 (+ 97005us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.761622 (+ 21961us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.810154 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.807978 24939 rpcz_store.cc:276] 0426 13:18:37.718898 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.045613 (+326715us) service_pool.cc:225] Handling call
0426 13:18:38.051051 (+  5438us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.051147 (+    96us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.141371 (+ 90224us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.699185 (+557814us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.801106 (+101921us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7040}
I20240426 13:18:38.811470 24944 rpcz_store.cc:276] 0426 13:18:37.794713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.178830 (+384117us) service_pool.cc:225] Handling call
0426 13:18:38.356260 (+177430us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.356412 (+   152us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.564843 (+208431us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.730058 (+165215us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.802529 (+ 72471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.812446 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.813334 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.813652 25145 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.813304 24944 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.814126 25145 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.510s	user 0.009s	sys 0.007s
I20240426 13:18:38.814575 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.817559 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:38.818099 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.818329 25157 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:38.818686 25157 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:38.819008 25157 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.819835 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.820586 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.821913 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.808704 25017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 49) took 1008 ms. Trace:
W20240426 13:18:38.824800 24921 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.825289 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:38.826303 24921 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.824985 25017 rpcz_store.cc:276] 0426 13:18:37.799904 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.569251 (+769347us) service_pool.cc:225] Handling call
0426 13:18:38.619407 (+ 50156us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.619652 (+   245us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.736743 (+117091us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.760376 (+ 23633us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.808021 (+ 47645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.761816 25087 rpcz_store.cc:276] 0426 13:18:37.665197 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.568975 (+903778us) service_pool.cc:225] Handling call
0426 13:18:38.596950 (+ 27975us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.597058 (+   108us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.610318 (+ 13260us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.738672 (+128354us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.739106 (+   434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.828996 25157 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.819490 25015 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 50) took 1003 ms. Trace:
I20240426 13:18:38.830808 25157 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.831357 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.830817 25015 rpcz_store.cc:276] 0426 13:18:37.815913 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.572571 (+756658us) service_pool.cc:225] Handling call
0426 13:18:38.595550 (+ 22979us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.627194 (+ 31644us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.697695 (+ 70501us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.819020 (+121325us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.819437 (+   417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1667840}
I20240426 13:18:38.838073 25018 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.838618 25013 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 52) took 1019 ms. Trace:
I20240426 13:18:38.839215 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.838984 25013 rpcz_store.cc:276] 0426 13:18:37.818927 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.648650 (+829723us) service_pool.cc:225] Handling call
0426 13:18:38.658044 (+  9394us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.664297 (+  6253us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.801616 (+137319us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.838188 (+ 36572us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.838567 (+   379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.840343 25149 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.842885 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 51) took 1024 ms. Trace:
I20240426 13:18:38.843271 25012 rpcz_store.cc:276] 0426 13:18:37.818517 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.586445 (+767928us) service_pool.cc:225] Handling call
0426 13:18:38.627182 (+ 40737us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.630232 (+  3050us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.727521 (+ 97289us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.842333 (+114812us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.842814 (+   481us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4608}
I20240426 13:18:38.831292 25157 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.847242 25095 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 57) took 1179 ms. Trace:
I20240426 13:18:38.847589 25095 rpcz_store.cc:276] 0426 13:18:37.667940 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.609099 (+941159us) service_pool.cc:225] Handling call
0426 13:18:38.610789 (+  1690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.610890 (+   101us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.767481 (+156591us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.846786 (+ 79305us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.847188 (+   402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.848778 25145 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.849534 25145 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.849836 25145 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.850534 25145 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.853025 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:38.857121 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:38.843807 24940 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.857630 25105 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:38.857694 24921 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.868252 25148 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.868983 25148 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.146s	user 0.008s	sys 0.003s
I20240426 13:18:38.868417 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:38.869544 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:38.870934 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.872145 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.872624 25170 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:38.873946 25170 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:38.874706 25170 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.875326 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 48) took 1016 ms. Trace:
I20240426 13:18:38.875711 24937 rpcz_store.cc:276] 0426 13:18:37.859193 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.803347 (+944154us) service_pool.cc:225] Handling call
0426 13:18:38.812912 (+  9565us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.812998 (+    86us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.846005 (+ 33007us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.874918 (+ 28913us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.875260 (+   342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.877697 25145 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.063s	user 0.005s	sys 0.003s
I20240426 13:18:38.882521 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.883675 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.883932 24941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 42) took 1026 ms. Trace:
I20240426 13:18:38.884352 24941 rpcz_store.cc:276] 0426 13:18:37.857225 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.643845 (+786620us) service_pool.cc:225] Handling call
0426 13:18:38.801616 (+157771us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.801728 (+   112us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.837918 (+ 36190us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.883453 (+ 45535us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.883813 (+   360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.886355 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 47) took 1027 ms. Trace:
I20240426 13:18:38.886881 24944 rpcz_store.cc:276] 0426 13:18:37.858890 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.812462 (+953572us) service_pool.cc:225] Handling call
0426 13:18:38.814152 (+  1690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.814236 (+    84us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.849563 (+ 35327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.885915 (+ 36352us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.886303 (+   388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.888217 24943 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 43) took 1030 ms. Trace:
I20240426 13:18:38.888633 24943 rpcz_store.cc:276] 0426 13:18:37.857553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.661992 (+804439us) service_pool.cc:225] Handling call
0426 13:18:38.693736 (+ 31744us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.714567 (+ 20831us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.838894 (+124327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.887738 (+ 48844us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.888162 (+   424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.889163 24944 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.881547 24945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 39) took 1025 ms. Trace:
I20240426 13:18:38.890983 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.890741 24945 rpcz_store.cc:276] 0426 13:18:37.856048 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.571894 (+715846us) service_pool.cc:225] Handling call
0426 13:18:38.605407 (+ 33513us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.605514 (+   107us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.776648 (+171134us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.861064 (+ 84416us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.881002 (+ 19938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.879767 25019 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 53) took 1060 ms. Trace:
I20240426 13:18:38.895174 25147 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.275s	user 0.014s	sys 0.018s
I20240426 13:18:38.894203 25019 rpcz_store.cc:276] 0426 13:18:37.856142 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.658082 (+801940us) service_pool.cc:225] Handling call
0426 13:18:38.731194 (+ 73112us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.731288 (+    94us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.811123 (+ 79835us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.860538 (+ 49415us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.879219 (+ 18681us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6823}
I20240426 13:18:38.898324 25145 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.900758 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.902603 25148 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.903857 25148 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.910308 25148 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.911708 25147 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:38.911263 25148 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.912457 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 50) took 1052 ms. Trace:
I20240426 13:18:38.913219 25145 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.914101 24946 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 44) took 1056 ms. Trace:
I20240426 13:18:38.914505 24946 rpcz_store.cc:276] 0426 13:18:37.857864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.666615 (+808751us) service_pool.cc:225] Handling call
0426 13:18:38.804131 (+137516us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.804262 (+   131us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.842703 (+ 38441us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.902070 (+ 59367us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.913781 (+ 11711us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.914046 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1014,"spinlock_wait_cycles":15265152}
I20240426 13:18:38.906599 24942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 45) took 1048 ms. Trace:
I20240426 13:18:38.908224 24943 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:18:38.908061 24854 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:38.918083 24942 rpcz_store.cc:276] 0426 13:18:37.858177 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.801473 (+943296us) service_pool.cc:225] Handling call
0426 13:18:38.819414 (+ 17941us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.819496 (+    82us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.854281 (+ 34785us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.906158 (+ 51877us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.906548 (+   390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.909647 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.886359 24941 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.907943 24938 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 41) took 1051 ms. Trace:
I20240426 13:18:38.913285 24940 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 49) took 1053 ms. Trace:
I20240426 13:18:38.920069 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.917714 25148 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.048s	user 0.001s	sys 0.004s
I20240426 13:18:38.920842 24938 rpcz_store.cc:276] 0426 13:18:37.856652 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.611570 (+754918us) service_pool.cc:225] Handling call
0426 13:18:38.800720 (+189150us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b539c 00007fda8d3b4f5b 00007fda962a76d9 00007fda951e4270 00007fda952c6651 00007fda952a3dc3 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2 00007fda8e853af1
0426 13:18:38.805314 (+  4594us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.805488 (+   174us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.880197 (+ 74709us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.907506 (+ 27309us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.907890 (+   384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19498880}
I20240426 13:18:38.912845 24937 rpcz_store.cc:276] 0426 13:18:37.859782 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.880621 (+1020839us) service_pool.cc:225] Handling call
0426 13:18:38.883276 (+  2655us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.883358 (+    82us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.890677 (+  7319us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.911831 (+ 21154us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.912386 (+   555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.923012 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.923900 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.925019 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.926208 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.926515 25018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 58) took 1068 ms. Trace:
I20240426 13:18:38.921234 24940 rpcz_store.cc:276] 0426 13:18:37.859505 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.842871 (+983366us) service_pool.cc:225] Handling call
0426 13:18:38.858222 (+ 15351us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.858802 (+   580us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.885829 (+ 27027us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.891729 (+  5900us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.912954 (+ 21225us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.913220 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22257408}
I20240426 13:18:38.926895 25018 rpcz_store.cc:276] 0426 13:18:37.857693 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.775706 (+918013us) service_pool.cc:225] Handling call
0426 13:18:38.838768 (+ 63062us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.838870 (+   102us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.879927 (+ 41057us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.922709 (+ 42782us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.926460 (+  3751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.923012 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 52) took 1018 ms. Trace:
I20240426 13:18:38.929945 24944 rpcz_store.cc:276] 0426 13:18:37.904693 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.888330 (+983637us) service_pool.cc:225] Handling call
0426 13:18:38.890557 (+  2227us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.890646 (+    89us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.906748 (+ 16102us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.922492 (+ 15744us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.922938 (+   446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.932478 24944 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.933271 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.925386 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.936635 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.938041 24939 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 46) took 1079 ms. Trace:
I20240426 13:18:38.938405 24939 rpcz_store.cc:276] 0426 13:18:37.858525 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.811587 (+953062us) service_pool.cc:225] Handling call
0426 13:18:38.821372 (+  9785us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.821592 (+   220us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.863982 (+ 42390us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.902357 (+ 38375us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.936623 (+ 34266us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.937987 (+  1364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30870400}
I20240426 13:18:38.946869 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 56) took 1041 ms. Trace:
I20240426 13:18:38.935052 25021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 57) took 1077 ms. Trace:
I20240426 13:18:38.948401 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.948212 25021 rpcz_store.cc:276] 0426 13:18:37.857411 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.769160 (+911749us) service_pool.cc:225] Handling call
0426 13:18:38.788183 (+ 19023us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.788312 (+   129us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.867791 (+ 79479us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.934564 (+ 66773us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.934997 (+   433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.958647 25145 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.959172 25145 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.061s	user 0.009s	sys 0.013s
I20240426 13:18:38.960166 24943 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 53) took 1055 ms. Trace:
I20240426 13:18:38.935849 25148 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.965799 24943 rpcz_store.cc:276] 0426 13:18:37.904983 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.889625 (+984642us) service_pool.cc:225] Handling call
0426 13:18:38.918253 (+ 28628us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.918348 (+    95us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.933646 (+ 15298us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.959647 (+ 26001us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.960091 (+   444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.935398 25147 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.968609 24938 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 57) took 1062 ms. Trace:
I20240426 13:18:38.969004 24938 rpcz_store.cc:276] 0426 13:18:37.906146 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.923593 (+1017447us) service_pool.cc:225] Handling call
0426 13:18:38.936639 (+ 13046us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.936747 (+   108us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.958036 (+ 21289us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.968121 (+ 10085us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.968542 (+   421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.970541 24945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 54) took 1065 ms. Trace:
I20240426 13:18:38.970975 24945 rpcz_store.cc:276] 0426 13:18:37.905271 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.892851 (+987580us) service_pool.cc:225] Handling call
0426 13:18:38.909107 (+ 16256us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952c6651 00007fda952a3dc3 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2 00007fda8e853af1
0426 13:18:38.909220 (+   113us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.909315 (+    95us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.945091 (+ 35776us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.970033 (+ 24942us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.970468 (+   435us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10339712}
I20240426 13:18:38.972115 25148 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.947249 24937 rpcz_store.cc:276] 0426 13:18:37.905858 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.923560 (+1017702us) service_pool.cc:225] Handling call
0426 13:18:38.925808 (+  2248us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.925894 (+    86us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.938871 (+ 12977us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.946392 (+  7521us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.946810 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.961813 25145 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.965438 25155 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.973376 25155 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.975497 25155 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.976500 25145 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.976795 25145 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.977093 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.977526 25145 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.960458 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 58) took 1054 ms. Trace:
W20240426 13:18:38.978317 24995 leader_election.cc:341] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.978482 24944 rpcz_store.cc:276] 0426 13:18:37.906444 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.931403 (+1024959us) service_pool.cc:225] Handling call
0426 13:18:38.933165 (+  1762us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.933249 (+    84us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.953818 (+ 20569us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.960001 (+  6183us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.960406 (+   405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.980715 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.981206 25145 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s	user 0.002s	sys 0.006s
I20240426 13:18:38.982242 25145 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:38.982257 24996 leader_election.cc:341] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.982887 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:38.984021 25155 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.986543 25147 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.987112 25147 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.076s	user 0.010s	sys 0.001s
I20240426 13:18:38.987656 24941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 51) took 1083 ms. Trace:
I20240426 13:18:38.988101 24941 rpcz_store.cc:276] 0426 13:18:37.904323 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.885497 (+981174us) service_pool.cc:225] Handling call
0426 13:18:38.920715 (+ 35218us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.920821 (+   106us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.961513 (+ 40692us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.987050 (+ 25537us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.987596 (+   546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.990537 24942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 55) took 1085 ms. Trace:
I20240426 13:18:38.990562 25147 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.991753 25147 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.992280 25147 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.993153 25147 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.991086 24942 rpcz_store.cc:276] 0426 13:18:37.905572 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.919184 (+1013612us) service_pool.cc:225] Handling call
0426 13:18:38.921842 (+  2658us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.922009 (+   167us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.954854 (+ 32845us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.983809 (+ 28955us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.990255 (+  6446us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.990484 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5518080}
I20240426 13:18:38.995005 25148 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.995123 25145 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.995541 25148 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.069s	user 0.011s	sys 0.012s
I20240426 13:18:38.995971 25155 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.996450 25155 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.998348 25155 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.998412 25148 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.999351 25148 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.999666 25148 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.999890 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.000552 25148 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:39.001170 24996 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.002254 25147 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.001s
I20240426 13:18:39.002501 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.003217 25147 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
W20240426 13:18:39.004426 24995 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.004999 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.005849 25148 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20240426 13:18:39.006906 25148 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.009493 25147 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.013036 25148 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.013547 25155 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.021522 25147 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.022022 25147 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.012s
I20240426 13:18:39.023977 25148 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.024493 25148 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.002s
I20240426 13:18:39.024854 25147 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.025617 25147 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.025907 25147 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.026670 25147 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.027623 25148 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.028415 25148 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.028787 25148 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.029227 25147 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240426 13:18:39.029678 25148 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.030246 25147 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.031438 25145 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.031921 25145 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.050s	user 0.005s	sys 0.017s
I20240426 13:18:39.032493 25148 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:18:39.033533 25148 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.034509 25145 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.035243 25145 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.035534 25145 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.036674 25147 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.036219 25145 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.038551 25145 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:39.039611 25145 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.040655 25148 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.045409 25145 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.057101 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:39.059366 25147 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.059463 25148 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.059885 25147 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.009s
I20240426 13:18:39.060199 25148 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.027s	user 0.007s	sys 0.014s
I20240426 13:18:39.062747 25147 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.062901 25148 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.063585 25147 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.063690 25148 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.063978 25147 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.064024 25148 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.064951 25145 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.065443 25145 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.001s
I20240426 13:18:39.064834 25147 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.064925 25148 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.067078 25147 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:18:39.068457 25147 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.067999 25145 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.068886 25145 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.069166 25145 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.069871 25145 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.070523 25148 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20240426 13:18:39.071511 25148 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.072081 25145 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.003s
I20240426 13:18:39.072903 25145 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.074620 25147 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.078373 25145 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.078459 25155 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.079080 25155 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.083289 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.083989 24956 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.085317 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.086015 25147 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.086454 25147 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20240426 13:18:39.086736 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.085907 25105 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.090493 25145 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.091064 25145 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
I20240426 13:18:39.087991 25155 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.091876 25155 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.092651 25155 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.092967 25155 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.094166 25148 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.094157 25145 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.095037 25145 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.095278 25145 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.095932 25145 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.098698 25145 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240426 13:18:39.099750 25145 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.104473 25155 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.106462 25145 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.106622 25147 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.107363 25147 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.107640 25147 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.108325 25147 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.109539 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.110065 24956 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.110132 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.110466 25147 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.024s	user 0.001s	sys 0.005s
I20240426 13:18:39.110736 25105 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.113968 25147 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.117724 25105 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.118468 25145 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.118917 25145 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.002s
I20240426 13:18:39.118777 24996 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.119697 25170 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.119819 25190 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.118815 24956 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.120213 25170 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.122236 25145 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.123148 25145 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.123497 25145 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.124341 25145 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.125242 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.125859 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.126641 25147 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.127193 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.120995 25155 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.127979 25157 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.128324 25157 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.128641 25157 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.130584 25145 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20240426 13:18:39.131491 25145 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.131531 25190 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:39.132632 25170 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.138370 25147 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.133075 25190 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.138818 25147 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.004s
I20240426 13:18:39.134474 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.141188 25147 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.141901 25147 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.138592 25145 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.142233 24921 leader_election.cc:341] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.142213 25147 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.122407 25148 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.143832 25148 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.073s	user 0.009s	sys 0.012s
I20240426 13:18:39.143949 25147 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.145356 25169 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.145766 25169 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.147686 25169 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.149708 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.150998 25070 leader_election.cc:341] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.142457 25157 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.153137 25147 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20240426 13:18:39.153836 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.154332 25031 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.160684 25147 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.161283 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.162408 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.163288 25170 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.167680 25147 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.169404 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.169744 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.170105 25157 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:39.172283 24921 leader_election.cc:341] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.173187 25145 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.173650 25145 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.042s	user 0.012s	sys 0.017s
W20240426 13:18:39.174160 25069 leader_election.cc:341] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.174660 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.176189 25169 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.176386 25145 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.177345 25145 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.177677 25145 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.178462 25145 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.179260 25170 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.180539 25145 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240426 13:18:39.180109 25170 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.181711 25145 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.182786 25147 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.183246 25147 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s	user 0.007s	sys 0.005s
I20240426 13:18:39.185091 25197 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.185572 25197 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.185971 25147 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.186764 25147 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.187091 25147 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.187391 25145 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.187878 25147 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.188385 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.190770 25147 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
W20240426 13:18:39.190721 24921 leader_election.cc:341] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.188781 25148 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.192148 25148 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.192550 25147 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.192528 25148 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.194869 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.195369 25197 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.195506 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.196743 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.196537 25148 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.198446 25147 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.198544 25197 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.199040 25197 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.199505 25197 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.206655 24852 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.208439 25145 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.208865 25145 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.015s
I20240426 13:18:39.208976 25147 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.209354 25147 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20240426 13:18:39.211633 25147 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.212231 25147 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.211843 25145 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.212495 25147 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.212601 25145 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.212888 25145 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.213148 25147 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.215050 25147 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240426 13:18:39.215451 25148 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.071s	user 0.005s	sys 0.002s
I20240426 13:18:39.215860 25147 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.216468 25148 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.215847 24851 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.213485 25145 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.220806 25145 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20240426 13:18:39.221727 25147 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.221968 25145 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.225143 25197 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.227339 25197 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.227505 25145 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.227862 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.229904 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.230405 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.231024 25148 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.234544 24921 leader_election.cc:341] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.235648 25190 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.236238 25190 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.238678 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.240095 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.240316 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.241425 24956 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:39.241255 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.246407 25190 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:39.247105 25197 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.247915 25197 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.250922 25197 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.250443 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.292766 25148 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.289958 25145 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.291677 25149 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240426 13:18:39.303798 25145 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.082s	user 0.023s	sys 0.016s
I20240426 13:18:39.303721 25148 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.087s	user 0.018s	sys 0.024s
I20240426 13:18:39.304236 25147 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.304695 25147 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.089s	user 0.017s	sys 0.012s
I20240426 13:18:39.306543 25145 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.306963 25147 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.256834 25190 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.307652 25147 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.306850 25148 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.307816 25190 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.307353 25145 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.290985 25169 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.308665 25031 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.310665 25148 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.311134 25148 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.311933 25148 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.312285 25169 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.313620 25148 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20240426 13:18:39.307947 25147 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.315210 25147 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.315562 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.317772 25148 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.317582 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.318046 25169 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:39.318832 25070 leader_election.cc:341] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.318503 25145 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.319857 25145 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.321429 24955 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:39.321911 25145 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20240426 13:18:39.322484 25169 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.324317 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.324859 25148 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.325687 25147 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20240426 13:18:39.325947 25169 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.326294 25169 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.326612 25147 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.326822 25169 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.326639 25190 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.328858 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.329188 25190 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.329558 25105 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:39.329587 25190 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.332399 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.332957 25147 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.332927 25105 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.334062 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.334889 25193 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.335269 25193 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.335558 25193 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:39.335845 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91252us
I20240426 13:18:39.337294 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.343394 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.344094 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.344612 25145 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.346203 25155 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.347292 25193 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.349409 25147 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.349814 25147 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.000s
I20240426 13:18:39.350580 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.351176 25105 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.351770 25148 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.352319 25193 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.352252 25148 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.000s
I20240426 13:18:39.353039 25170 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.353531 25170 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.356261 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.352239 25147 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.356868 25147 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.356858 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.357342 25147 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.356727 25148 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.357900 25148 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.358000 25190 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.358168 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.358353 25170 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.358255 25148 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.357919 25105 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.359035 25170 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.359400 25145 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.359460 25148 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.360152 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.360189 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.360975 25105 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.361074 25190 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.361831 25190 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:39.343591 25169 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.359355 25170 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.363091 25170 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.363554 25190 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.366281 25169 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.361920 25147 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.368602 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.368880 25147 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20240426 13:18:39.369890 25147 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
W20240426 13:18:39.369762 25070 leader_election.cc:341] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.370676 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.371266 24955 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.371812 25148 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20240426 13:18:39.373045 25148 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.378639 25147 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.385052 24851 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.390064 25170 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.392035 25147 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.393239 25147 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.001s
I20240426 13:18:39.393146 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.393790 25030 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.395757 25147 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.396561 25147 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.396826 25147 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.397449 25147 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.345810 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.419342 25190 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.420301 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.420907 25105 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.421468 25147 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.000s
I20240426 13:18:39.424789 25147 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.408603 24955 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:39.414525 25149 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.415807 25148 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.419440 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73751.1us
I20240426 13:18:39.427593 25149 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.430863 25105 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.431465 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.432359 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:39.433493 25157 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.403506 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.426863 25170 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.446695 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.447966 25169 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.448549 25169 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:39.449522 25169 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.444358 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.458228 24954 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.432354 24956 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:39.473086 25197 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240426 13:18:39.477197 24854 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.479605 25190 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.486850 25149 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.489054 24954 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.489773 25166 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.490182 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.490231 25166 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.485141 25157 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.492271 25145 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.494247 25145 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.150s	user 0.045s	sys 0.030s
I20240426 13:18:39.495848 25157 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.492786 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:39.498543 25166 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:39.501444 25070 leader_election.cc:341] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.504119 25145 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.504810 25145 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.505076 25145 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.505738 25145 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.506733 25149 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.507198 25149 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.507774 25145 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20240426 13:18:39.507627 25149 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.509805 25147 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.512012 25166 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.512588 25166 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.520676 25216 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.522079 25145 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.522433 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.524567 25197 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 13:18:39.521936 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.521644 25216 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.535089 25166 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.535807 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.536468 25216 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.537443 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.537478 25069 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.537950 25029 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.539119 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.539911 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.539121 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.541194 24921 leader_election.cc:341] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.544199 25216 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.545271 25216 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.545482 25030 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:39.545840 25149 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.547329 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.548610 25166 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.548713 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.549432 24954 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.568647 25148 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.569247 25148 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.196s	user 0.032s	sys 0.009s
I20240426 13:18:39.573658 25149 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.577349 24954 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:39.578584 25148 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.579224 25148 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.579322 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:39.579478 25148 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.580147 25149 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.580747 25149 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:39.580344 25148 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.581001 25190 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.579918 25166 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.581598 25149 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.576254 25216 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.585345 25145 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.590159 25148 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.002s
I20240426 13:18:39.592388 25148 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.606209 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:18:39.599524 25070 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.613294 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.618039 25216 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.619879 25216 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.627136 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.622081 25070 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:39.628458 24921 leader_election.cc:341] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.628849 25169 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.629231 25147 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.629804 25147 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.208s	user 0.014s	sys 0.023s
W20240426 13:18:39.629948 25070 leader_election.cc:341] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.631924 24851 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.632951 25147 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.633827 25147 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.634133 25147 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.634873 25147 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.636667 25190 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.638067 25147 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240426 13:18:39.639043 25147 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.639621 25148 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.637919 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.642266 25031 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.651718 25148 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.652192 25148 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.061s	user 0.008s	sys 0.010s
I20240426 13:18:39.654444 25147 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.654577 25148 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.655256 25148 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.655503 25148 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.656179 25148 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.657410 25031 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.658571 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.659694 25157 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.660171 25157 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.661013 25157 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.655622 25216 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.661938 25216 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.663695 25216 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.665010 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.665380 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.665639 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.665979 25104 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.666782 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.671437 25216 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.672816 25216 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.673177 25216 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.673485 25216 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.676005 25170 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.676774 25148 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20240426 13:18:39.676610 25170 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.677816 25148 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.686899 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.687713 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.689083 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.690099 25197 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.690544 25197 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.690956 25197 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.694931 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:39.696671 24921 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.698493 25148 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.704151 25170 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.706841 25147 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.707242 25147 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.068s	user 0.005s	sys 0.014s
I20240426 13:18:39.709653 25147 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.710301 25147 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.710602 25147 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.710728 24851 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:39.711870 25197 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.712917 25145 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.713438 25145 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.193s	user 0.014s	sys 0.007s
I20240426 13:18:39.714293 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.714833 25031 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.715818 25145 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.716414 25145 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.716507 25148 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.716660 25145 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.716881 25148 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.039s	user 0.009s	sys 0.010s
I20240426 13:18:39.719162 25148 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.719936 25148 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.720257 25148 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.719839 25216 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.720890 25148 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.721179 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.721647 24854 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.722493 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.723119 25031 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.723230 25148 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:39.722812 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.723886 25157 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.724191 25148 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.724404 25157 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.726843 25216 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.727350 25157 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.726189 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.723688 25145 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.728706 25105 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.730249 25145 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20240426 13:18:39.731209 25145 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.734900 25105 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.735890 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:39.736714 25216 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.737293 25216 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.738092 25216 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.741353 25148 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.744869 25197 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.745913 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:18:39.747123 24921 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.731951 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.760048 25190 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.760560 25190 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.764187 25157 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.764722 25157 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.782513 25145 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.782346 25147 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.783720 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.784755 25147 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.077s	user 0.017s	sys 0.019s
W20240426 13:18:39.785583 24995 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.787456 25157 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.785634 25147 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.791546 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.791878 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.792512 25105 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.793606 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:39.794327 25157 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.794734 25157 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.795089 25157 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.796149 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.796744 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
W20240426 13:18:39.797909 24921 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.797971 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:39.798882 25193 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.799224 25193 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.799286 25197 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.799707 25193 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.806793 25193 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.809283 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:39.810549 24995 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.814692 25147 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.823194 25147 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.823624 25147 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.038s	user 0.006s	sys 0.004s
I20240426 13:18:39.824625 25190 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.826086 25147 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.826941 25147 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.826702 25155 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.827224 25147 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.826720 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.827710 25105 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.832099 25193 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.827896 25147 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.833827 25157 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.834913 25147 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240426 13:18:39.837164 25147 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.837656 25145 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.838099 25145 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.107s	user 0.010s	sys 0.001s
I20240426 13:18:39.838466 24851 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.835065 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.840298 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.841456 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:39.842300 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.842545 25190 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.843011 25190 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:18:39.843449 24921 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.843753 25190 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.844617 25147 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.840411 25145 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.845438 25145 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.846638 25145 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.847651 25145 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.855610 25148 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.857729 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.859201 25148 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.135s	user 0.023s	sys 0.011s
I20240426 13:18:39.859341 25104 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.857059 25157 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.858750 25155 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:39.870215 25104 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.871680 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.872673 25157 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.873279 25157 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.874315 25157 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.874408 25031 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.893290 25145 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.055s	user 0.008s	sys 0.000s
I20240426 13:18:39.894847 25145 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.894457 25235 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:39.903239 25148 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.904124 25148 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.913378 25148 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.915511 25155 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.915956 25155 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.911024 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.920338 25155 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.924634 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.921937 25148 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.929010 25147 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.929495 25147 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.092s	user 0.027s	sys 0.025s
I20240426 13:18:39.929822 25148 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.000s
I20240426 13:18:39.930648 25148 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.933108 25157 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.934105 25147 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.934758 25147 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.935025 25147 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.935969 25145 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.935658 25147 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.939220 25235 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.939849 25235 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.942399 25235 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.944087 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.944698 25029 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.945242 25147 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20240426 13:18:39.945767 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:39.946117 25147 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.946739 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.944962 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.947714 25157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.949117 25157 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.949452 25157 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.949486 24995 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.948559 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.959249 25157 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.967674 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.968475 25029 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.970281 25147 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.972028 24854 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.979316 25148 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.982671 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.983428 25105 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.984706 25235 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.987731 25190 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.981370 25157 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.988319 25190 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.990728 25190 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.992883 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.993645 24956 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:39.995702 25029 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.996922 25149 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.997033 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.001345 25105 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.004864 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.006505 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.008394 25247 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.008453 25235 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.008891 25235 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.041021 25147 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.041471 25147 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.096s	user 0.004s	sys 0.008s
I20240426 13:18:40.001773 25157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.041290 25155 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:40.052523 25157 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.052981 25247 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.053735 25147 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.054438 25147 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.054754 25147 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.053982 25157 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.055342 25147 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.066810 25147 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20240426 13:18:40.067641 25147 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.072167 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.070062 25216 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240426 13:18:40.073866 25147 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:40.076319 24921 leader_election.cc:341] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.070597 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.064723 25235 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:40.085969 24921 leader_election.cc:341] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.088665 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.089362 25145 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.089773 25145 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.195s	user 0.019s	sys 0.003s
I20240426 13:18:40.092224 25145 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.092501 25166 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.092972 25145 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.093225 25145 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.093976 25145 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:40.095149 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:40.096447 25145 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240426 13:18:40.096957 25148 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.097347 25148 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.167s	user 0.010s	sys 0.018s
I20240426 13:18:40.097362 25145 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.099581 25148 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.100226 25148 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.102558 25148 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.103498 25148 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.108045 25166 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.109156 25166 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.113283 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.116235 25157 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.118233 25145 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.123868 25148 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20240426 13:18:40.125569 25148 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.117949 25166 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.127606 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.128383 25029 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:18:40.130872 25070 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:40.131994 25070 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.133627 25166 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.140956 24854 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.153517 25145 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.153821 25145 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.057s	user 0.016s	sys 0.004s
I20240426 13:18:40.154434 25148 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.158152 25166 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.159952 25166 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.160451 25145 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.161128 25145 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.161356 25145 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.161988 25145 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.165258 25145 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20240426 13:18:40.166500 25147 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.167305 25147 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.100s	user 0.013s	sys 0.020s
I20240426 13:18:40.169277 25147 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.169679 25147 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.169835 25147 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.171588 25145 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.173722 25148 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.174275 25148 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.049s	user 0.011s	sys 0.002s
I20240426 13:18:40.170375 25147 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.177080 25148 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.177819 25148 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.178121 25148 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.178856 25148 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.180938 25148 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:40.182272 25148 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.182942 25145 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.189955 25148 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.197451 25145 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.197883 25145 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s	user 0.009s	sys 0.008s
I20240426 13:18:40.199543 25148 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.177080 25147 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20240426 13:18:40.199944 25148 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.000s
I20240426 13:18:40.200626 25147 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.200331 25145 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.201045 25145 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.201328 25145 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.201993 25145 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.202204 25148 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.203003 25148 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.203265 25148 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.204205 25145 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20240426 13:18:40.204021 25148 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.206130 25145 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.206233 25147 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.206259 25148 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:40.207602 25148 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.215765 25148 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.216056 25235 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.217695 25145 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.222275 25216 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.224406 25148 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.224823 25148 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.000s
I20240426 13:18:40.227622 25148 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.228425 25148 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.228735 25148 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.230643 25148 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.233117 25148 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:18:40.233934 25148 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.234972 25216 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.238567 25145 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.239166 25145 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.011s
I20240426 13:18:40.239367 25148 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.241741 25145 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.243047 25145 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.243398 25145 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.245234 25155 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.245878 24954 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.244096 25145 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.254128 25148 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.254719 25148 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.006s
I20240426 13:18:40.256763 25216 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.257099 25145 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20240426 13:18:40.257381 25148 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.258276 25148 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.258544 25148 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.259325 25145 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.259191 25148 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.262038 25148 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240426 13:18:40.263146 25155 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.265477 25148 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.269491 25235 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.273002 25166 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.273459 25166 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.273967 25148 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.275761 25166 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.277621 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.278322 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.278780 25069 leader_election.cc:341] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.266660 25147 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.279320 25155 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.279592 25147 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.079s	user 0.013s	sys 0.000s
I20240426 13:18:40.279908 25155 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.282241 25155 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.282946 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.283324 25070 leader_election.cc:341] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.283737 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.284454 25155 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.284793 25166 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.284991 25155 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.287688 25145 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.287869 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.288908 25155 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.289151 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.289408 24996 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.289814 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.289300 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.290812 25101 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.290947 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.291798 25247 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.292518 25247 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.292876 25247 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.293177 25247 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.291838 25166 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.296702 25148 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.297284 25148 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.001s
I20240426 13:18:40.296636 25166 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.299728 25148 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.300072 25170 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.300443 25148 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.300817 25148 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.300691 25170 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.301524 25148 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.303001 25170 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.303795 25101 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.304272 25148 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20240426 13:18:40.303917 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.305157 25148 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.305778 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.306423 25166 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.308437 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.308607 25070 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.309033 25031 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.309672 25216 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.310089 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.310721 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.311580 25148 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.281973 25147 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.312723 25147 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.313007 25147 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.314709 25247 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.316347 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.320228 25170 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.320650 25170 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.321048 25170 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.326287 25235 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.332510 25029 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.340567 25247 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.341812 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.342665 25102 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.344521 25235 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.352427 25169 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.360929 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.361809 24956 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.363226 25145 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.363855 25145 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.105s	user 0.009s	sys 0.004s
I20240426 13:18:40.366466 25145 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.367336 25145 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.367652 25145 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.369256 25145 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.370864 25102 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.372366 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.373297 25247 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.373915 25247 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.374899 25247 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.373297 25145 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240426 13:18:40.383997 25145 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.341871 25170 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.395993 25145 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.404136 25147 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.407166 25147 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.127s	user 0.006s	sys 0.000s
I20240426 13:18:40.418309 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.419116 25029 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.429344 25029 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.430722 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.431866 25235 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.432476 25235 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.433981 25169 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.433517 25235 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.434443 25169 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.437198 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.437822 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.439149 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:40.440229 25166 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.440672 25166 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.441094 25166 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.444459 25169 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.445188 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.445757 25101 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.446707 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.447820 25170 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.447930 25216 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240426 13:18:40.447840 25070 leader_election.cc:341] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.453359 25166 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.456909 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.457659 24954 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.460290 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:40.461829 25070 leader_election.cc:341] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.463441 25166 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.470988 25148 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.471458 25148 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.166s	user 0.071s	sys 0.086s
I20240426 13:18:40.473783 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.474848 25148 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.475497 25148 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.475757 25148 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.476400 25247 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.476492 25148 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.476927 25247 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.478549 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.479365 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.480386 24995 leader_election.cc:341] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.480782 25197 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.481034 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.482650 25147 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.482702 25247 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.483410 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.483951 25166 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.484622 25166 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:18:40.484750 24996 leader_election.cc:341] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.485239 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.485932 25247 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.486687 25166 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.478554 25148 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20240426 13:18:40.488983 25145 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.489429 25145 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.106s	user 0.010s	sys 0.026s
I20240426 13:18:40.491372 25147 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.488981 25148 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.495651 24854 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.494333 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.500017 25145 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.500689 25145 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.500924 25145 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.501382 25145 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.505240 25145 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20240426 13:18:40.506090 25145 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.512100 25145 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.515201 25148 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.527693 25147 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.528123 25147 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.009s
I20240426 13:18:40.529464 24854 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.530334 25147 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.531018 25147 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.531286 25147 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.531916 25147 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.532783 25145 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.533210 25145 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.002s
I20240426 13:18:40.533890 25148 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.534101 25147 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:40.534300 25148 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.045s	user 0.017s	sys 0.000s
I20240426 13:18:40.534966 25147 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.535310 25145 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.535916 25145 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.536172 25145 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.536774 25145 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.537179 25148 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.537971 25148 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.538203 25148 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.538697 25145 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:18:40.538794 25148 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.539589 25145 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.541211 25148 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240426 13:18:40.542178 25148 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.546891 25148 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.547899 25147 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.553467 25145 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.567313 25145 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.567744 25145 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.000s
I20240426 13:18:40.568281 25147 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.568730 25147 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.003s
I20240426 13:18:40.568760 25148 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.569339 25148 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.027s	user 0.006s	sys 0.018s
I20240426 13:18:40.570180 25145 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.570852 25145 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.571142 25145 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.571043 25147 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.571944 25147 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.571805 25148 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.572466 25147 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.571868 25145 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.572777 25148 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.573135 25148 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.573174 25147 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.574008 25148 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.574471 25145 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.004s
I20240426 13:18:40.575902 25145 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.576424 25148 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20240426 13:18:40.577334 25148 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.578771 24854 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:40.581605 25145 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.584524 25148 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.583725 25147 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.001s
I20240426 13:18:40.587968 25235 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.588663 25235 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.591151 25235 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.594704 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.595407 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.596393 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.597126 25101 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.598973 25148 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.599481 25148 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.007s
I20240426 13:18:40.599443 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:40.601434 25235 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.601940 25235 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.602363 25235 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.602598 25148 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.603466 25148 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.603794 25148 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.604460 25148 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.606576 25148 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20240426 13:18:40.607470 25148 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.612520 25148 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.612823 25147 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.618116 25235 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.623744 25148 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.624320 25148 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.003s
I20240426 13:18:40.626701 25148 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.627389 25148 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.627736 25148 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.628538 25148 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.630847 25148 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:18:40.631542 25147 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.631778 25148 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.635936 25148 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.637858 25235 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.638482 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.639209 25031 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.641424 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.642028 25101 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.649012 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.654670 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.649497 25101 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.655723 25235 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.656247 25235 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.657054 25235 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.651461 25145 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.659993 25145 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.084s	user 0.033s	sys 0.026s
I20240426 13:18:40.660487 25148 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.660944 25148 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.013s
I20240426 13:18:40.652336 25147 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.662866 25147 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.050s	user 0.008s	sys 0.007s
I20240426 13:18:40.663483 25148 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.663467 25145 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.664425 25148 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.664600 25145 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.664758 25148 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.665048 25145 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.665513 25147 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.665513 25148 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.666410 25147 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.665863 25145 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.666756 25147 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.667559 25147 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.668136 25148 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240426 13:18:40.670262 25147 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:18:40.671156 25147 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.673017 25148 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.674952 25145 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.004s
I20240426 13:18:40.675976 25145 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.676662 25147 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.679224 25148 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.681105 25145 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.696202 25148 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.696691 25148 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.008s
I20240426 13:18:40.698279 25145 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.698673 25145 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.005s
I20240426 13:18:40.699081 25148 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.699746 25148 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.699985 25148 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.700385 25145 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.700906 25145 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.700593 25148 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.701121 25145 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.702044 25145 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.703759 25145 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
I20240426 13:18:40.705016 25145 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.706192 25148 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240426 13:18:40.707204 25148 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.707520 25147 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.707988 25147 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.037s	user 0.016s	sys 0.002s
I20240426 13:18:40.708178 25031 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.708798 25247 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.709250 25247 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.711102 25247 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.711061 25147 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.711992 25147 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.712319 25147 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.712833 25216 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.712994 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.713068 25147 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.713302 25216 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.713685 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.712419 25148 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.715209 25216 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.717391 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.717974 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.717880 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.718757 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.719357 25101 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.720111 25155 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.720230 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.720659 25155 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.720927 25104 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.721177 25155 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.723526 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:40.725760 25235 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.726233 24851 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.726261 25235 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.728690 25235 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.731634 25145 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.743014 25247 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.742836 25235 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.743891 25247 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.745754 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.746351 25029 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.747253 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.748961 24995 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.750042 25145 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.750458 25145 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.046s	user 0.008s	sys 0.000s
I20240426 13:18:40.752832 25145 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.753409 25145 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.753233 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.753708 25145 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
W20240426 13:18:40.754976 24996 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.755523 24996 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.756568 25190 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.758208 25247 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.759546 25145 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.762542 25155 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.768524 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.769228 24956 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.770103 25145 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20240426 13:18:40.763767 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.773460 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.774204 25216 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.774806 25216 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.775161 25145 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.775861 25216 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.776497 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.778789 25104 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.779661 25235 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.780457 25145 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.780407 25235 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.780920 25235 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.784471 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.785590 24921 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.786252 25166 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.786674 25166 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.788525 25166 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.790225 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.790747 25247 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.791532 25247 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.792484 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.790860 24955 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.794569 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:40.795434 25169 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.796039 25169 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.796254 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.797044 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.797930 25169 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.798154 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.798453 25169 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.798791 25169 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.798959 25190 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.799114 25169 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.799340 25190 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:18:40.799374 25070 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.799768 25190 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.801219 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.802757 25069 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.803586 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.804711 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.805542 24995 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.806213 25169 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.795470 25247 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.807937 25190 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.808789 25169 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.809175 25030 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:40.810133 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.811151 25101 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:40.813611 24921 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.814122 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.814723 25216 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:40.815850 24995 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.820947 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.823817 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.826807 25197 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.841540 25155 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.841269 25197 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.842921 25197 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.842355 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.827441 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.845081 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.845628 25155 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.845964 25155 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.846637 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.846760 25155 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.830765 25169 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.848351 25169 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.848853 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.849510 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.850068 25169 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.850795 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.851652 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.852803 25101 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.838618 25148 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.838416 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.856319 25148 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.149s	user 0.055s	sys 0.055s
W20240426 13:18:40.857108 25070 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.858335 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.858919 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.829435 25235 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.840507 25190 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.845877 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.859774 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:40.860002 25102 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.860265 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.765748 25147 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.057s	user 0.005s	sys 0.000s
I20240426 13:18:40.852206 25105 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.831787 25216 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.861809 25166 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.862783 25166 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.863168 25166 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.863862 25190 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.864322 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.864289 25190 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.864894 25169 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.840401 25104 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.866197 25190 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.866843 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.858659 25148 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.868152 25148 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.868407 25148 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.868782 25145 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.869158 25145 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.094s	user 0.027s	sys 0.013s
I20240426 13:18:40.868927 25148 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.870903 25148 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.002s
W20240426 13:18:40.869259 24995 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.871771 24955 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.851914 25170 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.872200 25170 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.872517 25170 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.872416 25166 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.872727 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:40.873947 25190 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.874536 25190 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.875064 25147 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.874946 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.875452 25190 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.877760 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.879205 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.879741 25029 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.880189 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.881001 25169 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:18:40.880203 24996 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.881500 25169 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:40.881644 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.882365 25247 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.882318 25169 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.884588 25148 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.885758 25247 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.889075 24853 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.878312 25170 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.890535 24956 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:40.891220 25170 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.878358 25166 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.895738 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.896451 25104 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:40.896198 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.878938 25102 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.871464 25145 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.898070 25145 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.898414 25145 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.899735 25145 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.902070 25145 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20240426 13:18:40.903087 25145 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.889575 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.904873 25148 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.905929 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:40.908339 25220 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.908814 25220 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:40.909550 25220 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.909706 25147 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.915239 25145 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.905994 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.928212 25170 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.927505 24851 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.929193 25170 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.936061 25147 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.936451 25147 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.062s	user 0.010s	sys 0.000s
I20240426 13:18:40.938715 25147 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.939385 25147 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.939663 25147 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.940951 25147 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.955168 25247 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.958863 25148 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.959290 25148 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.075s	user 0.012s	sys 0.004s
I20240426 13:18:40.961849 25148 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.962759 25148 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.963011 25148 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.963765 25145 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.964181 25145 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.061s	user 0.022s	sys 0.008s
I20240426 13:18:40.963882 25148 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.967762 25145 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.968475 25145 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.968811 25145 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.969276 25216 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.969336 25145 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.970815 25220 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.971239 25220 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.973232 25148 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20240426 13:18:40.975678 25145 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240426 13:18:40.977852 25148 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.977963 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:40.982308 25220 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.983132 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.984486 25069 leader_election.cc:341] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.985616 25147 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.000s
I20240426 13:18:40.989013 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.990278 25145 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:40.990602 25070 leader_election.cc:341] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.991036 25070 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.992112 25169 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.993522 25220 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.994648 25148 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.993909 25220 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.997761 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.997983 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.998490 25030 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:40.998916 25069 leader_election.cc:341] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.999449 25070 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.000044 25169 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.000386 25169 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.000695 25169 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.004128 25145 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.006382 25220 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.006752 25169 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.008138 25147 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.015027 25145 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.015422 25145 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.011s
I20240426 13:18:41.015779 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:41.017109 25069 leader_election.cc:341] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.028052 24853 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.149998 25145 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.150823 25145 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.151129 25145 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.152467 25145 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.150094 25220 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.028630 25169 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.028692 25155 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.058871 25247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.157166 25145 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.141s	user 0.003s	sys 0.005s
W20240426 13:18:41.134335 25070 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:41.157750 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.156664 25220 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.161954 25030 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.160249 25170 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:41.163118 25148 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.163347 25220 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.163549 25148 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.188s	user 0.008s	sys 0.021s
I20240426 13:18:41.164553 25147 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.164678 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.165861 25148 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.166312 25148 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.166509 25148 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.166937 25148 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.164925 25220 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.164746 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.169762 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:41.170423 25069 leader_election.cc:341] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.172910 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.158344 25155 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.174006 25149 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.174374 25149 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.174681 25149 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.176255 25170 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.180300 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:41.181798 24995 leader_election.cc:341] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.184167 25155 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.184710 25220 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.156323 25247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.185266 25220 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.189108 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:41.189196 25166 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.190804 25145 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.189824 25166 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.191942 25220 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.192759 25166 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.193660 25190 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.193248 25030 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.200814 25148 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.004s
I20240426 13:18:41.201457 25148 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.200080 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.188364 25149 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.200156 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.200376 25190 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.206504 24955 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.206490 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.214058 25247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.216058 25149 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.216748 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.217051 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.218024 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:18:41.218475 24996 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:41.219409 25069 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.219575 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.220068 25149 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.220495 25070 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:18:41.220712 25069 leader_election.cc:341] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.221297 25169 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.221776 25169 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.221993 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.222328 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.223088 24955 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.220417 25149 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.222579 25169 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.223393 24995 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.212329 25190 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.223816 24995 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.225813 25149 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.225719 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.227402 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.227919 25029 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.224505 25190 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:18:41.228291 25070 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.228588 25190 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.228675 25070 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.230979 25220 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.218827 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.233520 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.235281 25029 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:18:41.244524 25070 leader_election.cc:341] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.236011 25149 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.245149 24996 leader_election.cc:341] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.252104 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.251101 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.255431 25247 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.268959 25190 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.259950 25169 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.276728 25169 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.276414 24851 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.277536 25169 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.278508 25148 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.280805 25149 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.281342 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.282371 24955 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.283762 25145 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.287137 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.288573 24955 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:18:41.288695 25070 leader_election.cc:341] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.289389 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.290457 25149 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.290879 25149 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.293215 25149 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.297178 25147 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.297608 25147 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.290s	user 0.010s	sys 0.010s
I20240426 13:18:41.302521 25190 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.312075 25147 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.312657 25147 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.312880 25147 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.313819 25147 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.315676 25147 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.000s
I20240426 13:18:41.316519 25147 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.325455 25247 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.323011 24854 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.333613 24854 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.335285 25147 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.336196 25170 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.337585 25148 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.337365 25170 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.337911 25145 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.337977 25148 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.137s	user 0.021s	sys 0.017s
I20240426 13:18:41.338416 25145 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.148s	user 0.006s	sys 0.009s
I20240426 13:18:41.362159 25145 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.362854 25145 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.363107 25145 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.363652 25145 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.365806 25145 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.004s
I20240426 13:18:41.366384 25147 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.366732 25147 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.050s	user 0.009s	sys 0.003s
I20240426 13:18:41.367841 25145 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.369894 25147 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.370523 25147 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.370776 25147 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.371393 25147 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.373333 25147 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:41.374089 25147 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.373936 25148 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.374745 25148 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.375011 25148 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.376355 25148 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.379895 25147 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.382853 25169 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.384121 25170 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.384548 25170 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.386385 25170 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.387691 25148 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.000s
I20240426 13:18:41.388571 25148 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.391456 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.391571 25197 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.392086 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.392237 25197 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.394922 25197 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.395488 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.396275 25102 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.397286 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:41.398607 25235 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.398958 25235 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.399284 25235 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.403189 25145 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.405532 25235 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.407358 25235 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.408182 25148 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.408778 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.411063 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.411882 25102 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:41.422925 24954 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.430409 25169 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.435308 25145 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.435941 25145 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.068s	user 0.011s	sys 0.000s
I20240426 13:18:41.439132 25145 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.440057 25145 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.440449 25145 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.440939 25169 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.441399 25145 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.445016 25145 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20240426 13:18:41.446149 25145 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.452896 25145 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.469714 24921 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.468341 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:41.471462 25104 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.471305 25247 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.468051 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.477479 25220 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.479184 25029 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.489846 25147 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.490876 25147 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.117s	user 0.031s	sys 0.053s
I20240426 13:18:41.494334 25148 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.494783 25148 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.106s	user 0.016s	sys 0.028s
I20240426 13:18:41.497141 25148 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.498032 25148 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.498454 25148 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.501317 25148 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.516598 25148 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20240426 13:18:41.518278 25147 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.519646 25147 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.520025 25147 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.521679 25147 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.532348 25147 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.004s
I20240426 13:18:41.534694 25147 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.513137 25220 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.559059 25197 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.559645 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
W20240426 13:18:41.560634 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79614.5us
I20240426 13:18:41.546164 25220 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.562494 25155 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 13:18:41.570395 25220 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.573150 25104 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:41.575251 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:41.577991 25145 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.578459 25145 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.133s	user 0.016s	sys 0.043s
I20240426 13:18:41.582105 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.583252 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.583950 25028 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.579252 25340 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:18:41.585706 25069 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.585800 25340 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:41.591818 25070 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.593039 25220 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.593473 25220 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.593842 25220 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.594214 25148 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.603330 25145 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.603797 25145 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.603979 25145 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.604558 25145 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.610224 25345 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.616875 25345 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.618140 25145 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.000s
I20240426 13:18:41.619038 25145 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.622303 25147 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.600277 25340 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.620180 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.629910 25028 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.626375 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.619426 25345 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.633791 25148 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.634008 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.637895 25193 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.638334 25193 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.635381 25197 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.642714 25193 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.643332 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:41.644822 24995 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.647030 25220 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.650235 25145 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.652089 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:41.653506 24996 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.653900 24996 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.654701 25193 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.654850 25197 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.669916 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.670852 25147 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.671489 25147 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.137s	user 0.009s	sys 0.008s
W20240426 13:18:41.671371 25069 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.670024 25220 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.673110 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.673718 25028 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.673550 25147 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.674250 25147 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.674500 25147 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.674891 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:41.675395 25147 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.678347 25147 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20240426 13:18:41.679320 25147 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.679373 25220 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.680063 25028 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.681800 25145 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.681867 25148 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.682369 25145 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.063s	user 0.015s	sys 0.006s
I20240426 13:18:41.682454 25148 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.088s	user 0.016s	sys 0.010s
I20240426 13:18:41.684522 25147 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.684316 25070 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.684768 25145 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.685447 25145 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.685427 25149 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.685235 25148 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.686064 25148 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.686364 25148 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.686090 25149 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.687135 25148 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.687508 25149 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.692888 25148 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20240426 13:18:41.693986 25148 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.700773 25148 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.706382 25147 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.706805 25147 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.011s
I20240426 13:18:41.707319 25148 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.707930 25148 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20240426 13:18:41.709218 25147 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.710088 25147 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.710441 25147 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.710225 25148 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.711155 25148 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.711491 25148 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.711354 25147 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.712381 25148 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.714244 25148 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:18:41.715088 25148 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.717595 25147 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20240426 13:18:41.718703 25147 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.720316 25148 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.721779 25145 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.723853 25145 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.727021 25147 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:41.729597 25069 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:41.731248 25145 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.000s
I20240426 13:18:41.735184 25145 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.740248 25247 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.740855 25247 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.744524 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.746098 25145 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.747651 25148 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.748153 25148 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.033s	user 0.018s	sys 0.011s
I20240426 13:18:41.748531 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.749290 24955 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.750243 24995 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.751045 25155 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.750538 25148 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.751576 25148 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.750651 24853 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.751924 25148 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.752784 25148 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.743510 25247 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.757449 25155 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.757807 25149 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.761888 25148 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20240426 13:18:41.762806 25148 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.768337 25148 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.774160 25166 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.788384 25340 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.789005 25340 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.796998 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.791549 25340 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:41.809458 24921 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.823508 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.825392 25028 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.832304 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.833583 25340 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.833956 25340 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.834256 25340 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.834868 25148 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.835309 25148 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.073s	user 0.023s	sys 0.047s
I20240426 13:18:41.837270 25147 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.837606 25145 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.838047 25145 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.103s	user 0.032s	sys 0.011s
I20240426 13:18:41.837641 25147 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.119s	user 0.021s	sys 0.024s
I20240426 13:18:41.841795 25147 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.842518 25147 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.842763 25147 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.847433 25147 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.849298 25147 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20240426 13:18:41.840337 25145 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.849807 25145 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.850100 25145 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.850783 25145 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.854426 25147 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.853302 25340 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.855470 25149 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.856010 25149 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.857862 25340 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.858342 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.858711 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:18:41.859628 25069 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.859831 25145 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.000s
I20240426 13:18:41.859982 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.857564 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:41.860381 24921 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.860728 25030 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.858732 25149 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:41.861300 25070 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.862043 25070 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.861235 25148 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.863561 25148 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.863792 25148 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.864405 25169 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.864889 25147 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.866029 25148 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.867728 25169 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.868453 25148 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20240426 13:18:41.869320 25148 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.871038 25030 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:41.860788 25145 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.872087 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.872943 25340 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.873368 25340 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:41.874126 25340 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.879567 25148 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.880535 25155 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.881052 25155 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.887392 25166 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.887856 25166 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.889627 25166 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.891518 25149 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.892776 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.912459 25145 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.893164 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.893164 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.913138 25028 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.913393 25345 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.912743 24956 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.920789 25345 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.924237 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:41.925336 25149 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.929915 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.931906 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.932155 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.932643 25105 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
W20240426 13:18:41.933833 24921 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.925283 25345 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.935649 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.937670 25148 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.926049 25155 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.928503 25149 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.938803 25148 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.070s	user 0.010s	sys 0.005s
I20240426 13:18:41.939082 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.939657 25149 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.939868 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:41.942209 25155 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.947295 25149 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.947834 25148 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.948765 25148 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.949088 25148 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.949896 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.950630 24955 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.950261 25148 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.952414 25346 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.952621 25148 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
I20240426 13:18:41.952806 25346 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.953171 25346 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.953787 25148 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.964744 25147 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.965226 25147 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.111s	user 0.010s	sys 0.003s
I20240426 13:18:41.966993 24955 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.967974 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:41.968451 25148 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.968762 25220 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.969244 25220 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.970062 25220 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.975574 25169 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.976016 25169 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.977941 25169 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.978246 25149 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.979948 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.980535 25030 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.980707 25147 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.981421 25147 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.981573 25148 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.981685 25147 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.982097 25148 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.028s	user 0.008s	sys 0.007s
I20240426 13:18:41.982476 25147 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.984568 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.984457 25148 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.985200 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.986433 25148 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.986799 25031 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:41.986830 25069 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.986830 25148 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.987933 25070 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.988271 25148 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.989219 25220 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.989833 25220 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.990353 25220 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.990418 25148 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:18:41.990746 25030 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.991555 25148 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.993424 25346 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.994720 24854 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.995942 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.996500 25030 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.998220 25148 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.002625 25030 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.003635 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:42.004321 25340 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.004925 25340 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:42.005555 25346 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.005697 25340 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.006631 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.007457 25102 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.009397 25147 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.000s
I20240426 13:18:42.010257 25147 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.011759 25145 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.012217 25145 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.152s	user 0.010s	sys 0.002s
I20240426 13:18:42.014958 25145 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.015643 25145 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.016249 25145 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.016870 25145 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.019120 25145 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:18:42.021528 25145 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.020216 25220 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.027045 25220 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.028187 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.029044 25031 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.030889 25145 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.036489 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.038071 25069 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.041951 25031 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.042824 25145 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.043329 25145 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.012s
I20240426 13:18:42.043460 25070 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.044549 25220 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.045008 25220 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.045773 25220 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.047461 25169 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.047701 25102 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.047904 25169 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.049388 25147 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.049681 25169 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.051162 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.052510 25069 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.053475 25148 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.053937 25148 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.063s	user 0.026s	sys 0.026s
I20240426 13:18:42.053125 25145 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.055670 25340 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.055877 25145 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.056174 25340 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.056533 25148 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.056188 25145 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.057354 25148 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.057710 25148 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.058187 25340 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.058332 25148 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.058894 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.059578 25145 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.060786 25148 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.004s
I20240426 13:18:42.062011 25148 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
W20240426 13:18:42.061908 24921 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.062913 25145 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20240426 13:18:42.062095 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.065996 25145 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.066562 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.067206 25102 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:42.053292 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.068027 25148 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.068087 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.069275 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.072872 25169 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.073249 25169 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.073614 25169 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.076699 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.077541 25343 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.078012 25343 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.078388 25343 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.082471 25169 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.087719 25147 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.088183 25147 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.078s	user 0.010s	sys 0.001s
I20240426 13:18:42.092334 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.093711 25069 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.094558 25169 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.097895 25170 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.100834 25145 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.111560 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.113658 25145 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.113716 25147 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.125309 25147 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.125656 25147 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.126230 25147 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.127727 25147 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.001s
I20240426 13:18:42.117465 24854 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.120293 25148 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.129598 25148 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.068s	user 0.034s	sys 0.019s
I20240426 13:18:42.132056 25148 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.132773 25148 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.132269 25220 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.133152 25148 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.133811 25148 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.125241 25031 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.135326 25148 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:18:42.145462 25145 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.080s	user 0.012s	sys 0.004s
I20240426 13:18:42.141443 25147 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.146385 25148 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.124555 25343 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.151275 25147 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.151396 25343 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.152107 25148 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.153475 25155 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.153942 25155 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.155964 25145 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.156526 25155 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.156672 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.156652 25145 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 13:18:42.158128 24995 leader_election.cc:341] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.158497 25145 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.159132 25145 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.160005 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:42.163219 24921 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.165050 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.164501 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.165503 25145 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.001s
I20240426 13:18:42.166436 25145 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.168396 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.165845 25102 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.175496 25169 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.176388 25169 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.179622 25169 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.183950 25145 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.186081 25343 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.198808 25147 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.199409 25147 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.058s	user 0.006s	sys 0.022s
I20240426 13:18:42.201025 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.201858 25104 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.202441 25147 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.203296 25147 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.203660 25147 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.207271 25147 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.209010 25147 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20240426 13:18:42.209251 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.209829 25147 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.210739 25247 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.211208 25247 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.211616 25247 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.220145 25155 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:42.220327 25069 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.220636 25155 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.222529 25155 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.226691 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.227596 24955 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.230598 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:42.231070 25104 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.232156 25155 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.232525 25155 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.232860 25155 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.235247 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.235819 25031 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.238054 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.238787 25343 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.239229 25343 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:42.240025 25343 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.241991 25155 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.235889 25102 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.247160 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.247696 24954 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.248675 25247 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.250690 25148 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.251137 25148 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.105s	user 0.012s	sys 0.023s
I20240426 13:18:42.255249 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.256448 24995 leader_election.cc:341] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.256765 25147 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.257217 25148 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.257908 25148 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.258177 25148 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.258239 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.258409 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.258806 25102 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.258844 25148 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.259748 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:42.260898 25193 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.261245 25148 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20240426 13:18:42.262107 25148 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.265290 25220 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.265780 25220 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.266678 25155 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.268752 25148 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.269140 25220 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.270236 25193 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.270690 25247 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.271301 25193 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.275702 25145 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.276152 25145 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.110s	user 0.012s	sys 0.000s
I20240426 13:18:42.276021 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.276695 25030 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.278070 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:42.278482 25145 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.279129 25145 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.279451 25145 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.280460 25145 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.282922 25145 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:42.282756 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.283465 25104 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.283799 25145 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.289796 25102 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.281190 25220 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.291026 25220 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.291409 25220 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.293951 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.295585 25388 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.295960 25388 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.296582 25388 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.297996 25220 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.290449 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.286182 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.299894 25220 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.316653 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.319972 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.320628 25029 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.313388 24954 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.379456 25388 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.379976 25388 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.381886 25388 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.378427 24854 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.384766 25147 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
W20240426 13:18:42.389254 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:42.390450 25147 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.181s	user 0.012s	sys 0.032s
I20240426 13:18:42.388620 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.396808 24995 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.399363 25104 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.405560 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.406803 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.410882 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.415786 25145 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.418313 24996 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.418726 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.419426 25388 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.423183 25029 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.427503 25170 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.429337 25340 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.429474 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.430265 25220 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.430742 25220 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.433621 25147 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.434194 25147 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.434392 25147 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.431555 25220 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.435051 25147 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.433614 25102 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.440763 25147 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.046s	user 0.004s	sys 0.000s
I20240426 13:18:42.441574 25147 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.448122 24954 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.451390 25148 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.451999 25148 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.190s	user 0.008s	sys 0.026s
I20240426 13:18:42.456977 25148 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.457617 25148 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.457887 25148 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.458498 25148 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.459916 25148 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240426 13:18:42.460772 25148 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.463085 25169 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.478292 25170 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:42.481338 25147 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.477461 25220 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240426 13:18:42.502204 25169 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.503084 25148 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.525061 25170 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.526011 25169 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.545311 25145 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.545707 25145 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.262s	user 0.005s	sys 0.014s
I20240426 13:18:42.547742 25145 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.559172 25145 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.559417 25145 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.568117 25147 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.551628 24851 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.570798 25147 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.129s	user 0.012s	sys 0.005s
I20240426 13:18:42.573069 25147 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.573693 25147 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.573977 25147 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.573747 25145 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.576262 25148 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.576714 25148 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.116s	user 0.012s	sys 0.003s
I20240426 13:18:42.577303 25145 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.002s
I20240426 13:18:42.578153 25145 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.578876 25148 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.579551 25148 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.579789 25148 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.585341 25147 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.587152 25147 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I20240426 13:18:42.587667 25148 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.589743 25148 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20240426 13:18:42.590482 25148 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.592473 25147 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.595963 25247 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.596410 25247 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.598287 25247 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.599714 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.601848 25147 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.602702 24995 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.600826 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.604691 25148 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.605141 24996 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.605551 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.606241 25247 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.611574 25145 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.612375 25147 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.612780 25147 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.000s
I20240426 13:18:42.614809 25147 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.615427 25147 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.615693 25147 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.616461 25147 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.617313 25340 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.617655 25145 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.618113 25145 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.040s	user 0.009s	sys 0.001s
I20240426 13:18:42.617719 25340 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.618724 25147 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:42.619787 25147 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.620362 25145 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.621140 25145 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.621428 25145 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.621992 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.622085 25145 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.625087 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.625473 25148 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.625854 25148 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.004s
I20240426 13:18:42.626736 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.627389 25170 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.627821 25340 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.628650 25148 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.629297 25148 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.629546 25148 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.630064 25148 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.631742 25145 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20240426 13:18:42.632575 25145 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.632690 25148 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:18:42.634799 25147 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.637684 25148 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.638214 25145 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.642699 25148 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.648464 25345 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.656404 25148 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.656776 25148 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.008s
I20240426 13:18:42.678148 25147 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.678669 25147 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.059s	user 0.009s	sys 0.010s
I20240426 13:18:42.679872 25247 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.681314 25148 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.681947 25148 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.682173 25148 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.682701 25148 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.684829 25145 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.684939 25148 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.002s
I20240426 13:18:42.685333 25145 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.015s
I20240426 13:18:42.685746 25148 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.687582 25145 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.688306 25145 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.689172 25145 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.692123 25145 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.689705 25147 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.693814 25147 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.694247 25147 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.690981 25148 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.697022 25147 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.707363 25148 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.696000 25145 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20240426 13:18:42.707798 25148 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.009s
I20240426 13:18:42.708099 25147 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
W20240426 13:18:42.708524 24921 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.717914 25155 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.713203 24954 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.726315 25147 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.728564 24955 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.714313 25145 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.732466 25147 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.732704 25155 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.733124 25155 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.734683 25145 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.735765 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.710012 25148 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.736398 25148 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.736658 25148 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.737105 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:42.737264 24995 leader_election.cc:341] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.738154 25103 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.737259 25148 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.740419 25148 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.000s
I20240426 13:18:42.740173 24996 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.741287 25148 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.741178 25169 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.741336 25193 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.741942 25193 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.742373 25193 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.736979 25155 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:42.750046 24995 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.757997 25148 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.774295 25169 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.796499 25166 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.797024 25166 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.799322 25166 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.804931 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.805737 25027 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.811236 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:42.812707 25149 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.813184 25149 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.813633 25149 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.822923 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.823794 24954 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.826689 25193 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.831082 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.832201 24995 leader_election.cc:341] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.831768 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.832847 25103 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.844692 25155 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.849211 25149 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.853253 25145 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.859781 25145 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.146s	user 0.022s	sys 0.025s
I20240426 13:18:42.862677 25145 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.863314 25145 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.863569 25145 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.858209 25247 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.864395 25247 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.866407 25247 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.867038 25145 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.867658 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.867988 25147 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.868453 25147 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.142s	user 0.044s	sys 0.074s
W20240426 13:18:42.869027 24995 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.869694 25145 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20240426 13:18:42.870597 25145 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.870729 25147 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.867337 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.871407 25147 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.871726 25147 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
W20240426 13:18:42.872553 24996 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.873278 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.874577 25247 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.875366 25193 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.874835 25103 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.880412 24996 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.881353 25388 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.881561 25193 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.882057 25193 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.881816 25388 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.882912 25193 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.884114 25388 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.884737 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.884989 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.884881 25147 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.885890 25102 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
W20240426 13:18:42.886152 24995 leader_election.cc:341] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.883878 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.887027 25147 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.018s	user 0.002s	sys 0.002s
I20240426 13:18:42.887074 24955 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.887936 25147 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.888549 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.889407 25422 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.889755 25422 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.890048 25422 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.893249 25147 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.894179 24955 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.895392 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:42.896301 25169 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.896703 25169 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.897462 25169 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.902026 25148 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.903872 25147 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.904832 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.880026 25145 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.907757 25148 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.167s	user 0.017s	sys 0.047s
I20240426 13:18:42.906661 25149 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.907289 25147 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20240426 13:18:42.907306 25030 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.910032 25148 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.910648 25148 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.910930 25148 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.911548 25148 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.913761 25148 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:18:42.914225 25030 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.914525 25148 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.917009 25147 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.917619 25147 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.917881 25147 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.918433 25147 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.919829 25148 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.922006 24853 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.923218 25147 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20240426 13:18:42.923871 25147 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.925055 25422 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.932310 25422 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.934187 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.934830 25103 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.935468 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.942268 25145 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.942744 25145 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.072s	user 0.008s	sys 0.001s
W20240426 13:18:42.945497 24995 leader_election.cc:341] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.945681 24955 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.948139 25148 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.948766 25148 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.000s
I20240426 13:18:42.955313 25145 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.955924 25145 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.956212 25145 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.956247 25148 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.956813 25145 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.957423 25148 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.958297 25145 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20240426 13:18:42.960556 25148 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.961217 25148 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.962433 25103 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.962740 25145 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.963585 25148 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20240426 13:18:42.965626 25148 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.966468 25193 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.966888 25193 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.968623 25193 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.972433 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.973604 24995 leader_election.cc:341] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.973292 24852 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.976225 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.977691 25147 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.979400 25169 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.982741 25388 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.983893 25388 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.984710 25388 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.986639 25148 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.987769 25169 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.993089 25145 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.009322 25147 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.009896 25147 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.086s	user 0.006s	sys 0.004s
I20240426 13:18:43.016266 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.018100 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.020421 25145 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.020999 25145 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.058s	user 0.007s	sys 0.003s
I20240426 13:18:43.023833 25147 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.024509 25145 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.025193 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.025653 25145 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.025918 25145 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.026477 25147 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.026810 25147 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.027235 25388 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.030396 25388 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.030865 25388 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.027407 25147 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.028231 25145 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.038008 25149 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.038460 25149 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.040341 25149 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.042022 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.042692 25030 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.036590 25147 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20240426 13:18:43.042881 25145 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s	user 0.002s	sys 0.004s
I20240426 13:18:43.042876 25388 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.044132 25145 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.044098 25070 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:43.044670 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.045048 25149 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.045384 25149 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.045711 25149 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.045835 25069 leader_election.cc:341] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.046787 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.048194 24995 leader_election.cc:341] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.049372 25388 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.049832 25145 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.050958 25147 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.051751 25149 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.054553 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.056191 25069 leader_election.cc:341] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.057673 25149 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.058297 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.058907 25029 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.066790 25145 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.067315 25145 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.002s
I20240426 13:18:43.070844 25343 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.071413 25343 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.073390 25343 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.070014 25145 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.074167 25145 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.074455 25145 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.075165 25145 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.075213 25029 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.077419 25145 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20240426 13:18:43.077491 25070 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.078038 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.078270 25149 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.078373 25145 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.078830 25149 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.079916 25166 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.079666 25149 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.080379 25166 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.082283 25166 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.082599 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.082917 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.083173 25103 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.084200 25069 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.084156 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.085080 25030 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.086233 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.087455 25166 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.087852 25166 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.088277 25166 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.089370 25145 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.092857 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.093911 25104 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:43.095613 25147 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.099838 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.100406 25145 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.101253 25145 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.004s
I20240426 13:18:43.101192 25346 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.104140 25343 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.104632 25343 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.104698 25145 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.105666 25145 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.106005 25145 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.106688 25343 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.106822 25145 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.107327 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.109674 25145 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240426 13:18:43.110863 25145 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.100755 25166 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.115630 24854 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.117048 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.118796 25069 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.120074 25166 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.121078 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.121654 25030 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.128545 25030 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.128700 25145 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.129575 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.130669 25166 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.131143 25166 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.131911 25166 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.133939 25346 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.139037 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.141551 25145 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.142355 25145 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.032s	user 0.013s	sys 0.007s
I20240426 13:18:43.148507 25145 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.153292 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.163915 25145 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.164498 25145 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.164817 25388 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.165309 25388 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.165521 25145 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.166146 25388 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.168188 25148 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.168642 25148 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.203s	user 0.040s	sys 0.031s
I20240426 13:18:43.168301 25145 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20240426 13:18:43.175088 25148 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.177415 25148 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.177651 25247 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.178167 25145 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.178094 25247 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.177698 25148 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.179976 25247 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.181398 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.184530 25145 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.181303 25148 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.185588 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.187520 25148 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.001s
I20240426 13:18:43.189479 25148 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.190570 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.194870 25148 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.196739 25145 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.197263 25145 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.002s
I20240426 13:18:43.200109 25145 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.201192 25145 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.201586 25145 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.202463 25145 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.202986 24852 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.204787 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.211189 25170 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.211925 25170 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.212508 25247 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.214689 25247 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.215692 25170 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.213263 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.216538 25247 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.219537 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.220853 24995 leader_election.cc:341] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.220727 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.223505 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.224174 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.225090 25104 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.228819 25145 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.001s
I20240426 13:18:43.229776 25145 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.237238 24954 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.238757 25145 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.240180 25343 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.242811 25147 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.243391 25147 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.193s	user 0.044s	sys 0.030s
I20240426 13:18:43.249898 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.249933 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.252187 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.253333 25247 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.253901 25247 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.254215 25247 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.254511 25247 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.260174 25147 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.260936 25147 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.261242 25147 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.264398 25166 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.265038 25166 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.265427 25247 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.266041 25147 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.267696 25166 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.268460 25147 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.003s
I20240426 13:18:43.268756 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.269142 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.270327 24995 leader_election.cc:341] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.270184 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.270414 25070 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.271579 25247 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.271522 24921 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:43.272337 25069 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.272783 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.273792 25166 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.277287 25147 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.280949 25170 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.281755 25166 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.282235 25166 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.283843 25029 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.289721 25166 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.290405 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.295082 25069 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.292549 25145 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.292769 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.296679 25147 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.297076 25145 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.067s	user 0.019s	sys 0.006s
I20240426 13:18:43.297024 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.301064 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.301245 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.300055 25145 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.304327 25102 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.307796 25145 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.308281 25145 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.309265 25148 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.309927 25148 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.121s	user 0.036s	sys 0.045s
I20240426 13:18:43.309240 25145 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.306284 25166 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.312208 25166 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.312570 25166 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.313241 25145 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20240426 13:18:43.314337 25145 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.315069 25148 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.315747 25148 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.316002 25148 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.316589 25148 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.318992 25148 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20240426 13:18:43.320574 25148 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.321081 25145 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.326591 25148 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.328107 25166 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.331084 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:43.331413 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.332201 25030 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.332669 25069 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.337765 25102 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.343178 25148 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.343693 25148 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.006s
I20240426 13:18:43.345180 25145 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.345890 25145 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.004s
I20240426 13:18:43.348712 25145 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.349607 25145 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.349992 25145 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.350798 25145 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.354988 25145 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.001s
I20240426 13:18:43.352182 25148 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.356284 25148 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.356703 25148 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.357539 25148 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.343092 24854 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.358803 25145 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.357533 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.362203 25166 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.362831 25148 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.004s
I20240426 13:18:43.364857 25388 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.365327 25388 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.366207 25388 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.365379 25148 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.363054 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.371930 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.382025 25247 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.378944 25166 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.384305 25166 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.385373 25166 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.386198 25247 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.411201 25247 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.411837 25145 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.415200 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.417309 25148 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.411700 25247 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.433965 24852 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.442292 25147 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.442688 25147 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.168s	user 0.037s	sys 0.059s
I20240426 13:18:43.443614 25247 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.447784 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.451277 24995 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.454427 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.457216 25147 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.457945 25147 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.458212 25147 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.458858 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.458890 25147 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.460104 25388 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.461513 25147 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20240426 13:18:43.457407 25169 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:43.463943 24996 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.464426 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.465341 25388 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.466833 25147 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.471967 25170 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.472435 25170 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.474220 25170 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.475158 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.475803 25102 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:43.476337 25149 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.481870 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:43.482648 25340 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.482991 25340 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.483301 25340 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.490619 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.491314 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:43.492201 25145 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.492317 25148 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.492673 25145 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.137s	user 0.015s	sys 0.005s
I20240426 13:18:43.493291 25148 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.128s	user 0.013s	sys 0.009s
I20240426 13:18:43.494983 25145 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.495694 25148 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.496378 25148 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.495635 25145 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.496626 25148 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.496938 25145 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.503412 25148 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.506856 25148 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20240426 13:18:43.512923 25148 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.526500 25145 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.535035 25145 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.001s
I20240426 13:18:43.535688 25145 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.539978 25145 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.541980 25149 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.547727 25340 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.560426 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57189.8us
I20240426 13:18:43.560401 25388 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.574793 25148 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.575044 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:43.582342 25340 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.582904 25220 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.569615 25388 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.584406 25220 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.600749 25220 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.600765 25388 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.626236 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.627065 25023 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.621696 25457 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20240426 13:18:43.635311 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.638132 25149 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:43.638381 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.638553 25149 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.638981 25069 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.639286 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.641469 25149 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.644457 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.645193 25147 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.645061 25103 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.649520 25145 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.649973 25145 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.114s	user 0.021s	sys 0.020s
I20240426 13:18:43.652129 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.652477 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.652722 25105 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.692116 25148 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.693367 25148 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.181s	user 0.016s	sys 0.022s
I20240426 13:18:43.698763 25145 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.699433 25145 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.700107 25145 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.690934 25193 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.708935 25145 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.709827 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.694526 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58946.1us
I20240426 13:18:43.719240 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:43.708782 25193 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.728883 25148 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.729614 25148 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.729880 25148 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.730433 25148 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.732911 25193 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.740386 25145 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.090s	user 0.004s	sys 0.016s
I20240426 13:18:43.743438 25145 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.740538 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.734037 25023 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:43.742260 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:43.748308 25148 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.055s	user 0.005s	sys 0.001s
I20240426 13:18:43.756583 25148 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
W20240426 13:18:43.752993 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59912us
I20240426 13:18:43.757264 25029 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.759464 25346 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.760121 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:18:43.756211 25070 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.762665 25346 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:43.768826 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.782189 25145 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.767239 25069 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.784906 25469 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.785822 25346 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.786357 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.787457 25340 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.788965 25469 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.776912 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.796224 25340 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.798624 25340 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.800470 25466 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.800793 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.804319 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.805087 25023 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:43.805917 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.807273 25170 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.808605 25147 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.809031 25147 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.342s	user 0.021s	sys 0.052s
I20240426 13:18:43.811282 25147 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.811950 25147 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.812340 25147 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.812939 25147 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.800117 25469 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.799050 24996 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:43.814049 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52370.7us
I20240426 13:18:43.817289 25147 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:18:43.826552 25147 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.827011 25468 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:43.829603 24852 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:43.831777 24995 leader_election.cc:341] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:43.833313 24995 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.833851 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.835572 25155 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.845559 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.849954 25466 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.850564 25466 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.851121 25466 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.855835 25148 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.865031 25170 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:18:43.871171 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.884804 25147 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.890708 25169 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:43.905288 25069 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.907903 25388 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.925091 25148 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.925575 25148 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.169s	user 0.009s	sys 0.007s
I20240426 13:18:43.934667 25148 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.935341 25148 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.927335 25469 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.937336 24995 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.937630 25469 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.938962 25145 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.939409 25145 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.196s	user 0.024s	sys 0.022s
I20240426 13:18:43.935580 25148 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.942118 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:43.943434 25148 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.945951 25148 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.001s
I20240426 13:18:43.947440 25148 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.953025 25145 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.952803 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.954293 25145 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.954643 25104 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.959250 25145 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.962679 25104 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.959928 25145 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.964603 25220 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.968847 25145 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20240426 13:18:43.970192 25145 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:43.966300 24995 leader_election.cc:341] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.951889 25466 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.979049 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.980113 25469 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.980737 25469 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.983258 25169 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.985306 25476 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.985920 25476 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.988385 25476 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.981571 25469 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.025846 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.026504 25025 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.033120 25145 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.039769 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.044103 25148 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.046036 25147 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.046413 25147 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.221s	user 0.013s	sys 0.006s
I20240426 13:18:44.052289 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:44.055150 25476 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.055516 25476 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.056268 25476 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.057085 25147 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.057709 25147 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.057911 25147 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.058398 25147 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.058954 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.066107 25466 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.070221 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.078383 25023 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.074592 25476 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.075081 24853 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:44.061372 25069 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.085214 25069 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.088146 24956 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.095115 25148 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.095729 25148 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.148s	user 0.010s	sys 0.000s
I20240426 13:18:44.099848 25469 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.100349 25469 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.077260 25147 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.031s	user 0.003s	sys 0.002s
I20240426 13:18:44.101970 25147 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.103034 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.104373 24995 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.105665 25469 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.108811 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.109436 25145 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.109469 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.110006 25145 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.140s	user 0.008s	sys 0.001s
I20240426 13:18:44.109854 25148 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.110812 25148 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.091506 25023 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.111435 25148 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.112100 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.112848 25220 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.112630 25148 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.113301 25220 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.114078 25220 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.114902 25148 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20240426 13:18:44.115773 25148 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.116256 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.117322 25476 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.117911 25388 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.118274 25388 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.118583 25388 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.120406 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.120968 24954 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.126757 24954 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.127559 25148 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.128356 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.128439 25343 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:18:44.132144 25023 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.128659 25388 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.132305 25343 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.135627 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.135892 25343 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.136283 25104 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.138972 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.139794 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.140642 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:18:44.141271 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.141433 25149 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.141892 25149 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.142803 25346 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:44.143616 25149 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.144125 25169 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.144245 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.144623 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.147641 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.145946 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.147612 25169 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.148761 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:44.149644 25149 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.146441 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.149955 25149 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.149921 25169 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.150496 25149 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.150336 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.151762 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:44.152599 25469 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.153085 25469 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:44.153349 25145 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.153849 25469 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.154603 25069 leader_election.cc:341] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.156533 24995 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.156461 25149 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.153993 25145 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.158223 25145 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.158839 25145 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.159154 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.159684 25025 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.160766 25388 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.162973 25147 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.166399 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.167341 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:44.167341 25145 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.000s
I20240426 13:18:44.168090 25220 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.172191 25145 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.172272 25220 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.173105 25220 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.176745 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.178095 25069 leader_election.cc:341] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.168404 25149 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.179697 25145 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.181077 25148 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.181524 25148 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.001s
I20240426 13:18:44.184593 25343 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.179822 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.197070 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:44.199381 25148 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.202534 25148 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.202804 25148 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.203717 25148 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.206888 25148 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.001s
I20240426 13:18:44.207747 25148 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.207835 25147 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.208339 25147 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.107s	user 0.011s	sys 0.005s
I20240426 13:18:44.209322 25346 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.210690 25147 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.211360 25147 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.211614 25147 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.212205 25147 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.214160 25147 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I20240426 13:18:44.215096 25147 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.214197 25148 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.220003 25147 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.224470 24853 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.226548 24853 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.233242 25147 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.231274 25148 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.231062 25145 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.235157 25145 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.063s	user 0.006s	sys 0.010s
I20240426 13:18:44.237413 25145 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.238093 25145 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.238333 25145 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.239094 25145 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.241173 25145 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:44.241990 25145 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.244344 25148 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.001s
I20240426 13:18:44.245644 25343 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.246137 25343 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.247035 25145 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.247967 25343 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.247848 25148 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.248560 25148 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.248833 25148 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.249415 25148 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.248337 25147 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.000s
I20240426 13:18:44.252068 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.251014 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.268060 25147 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.268769 25147 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.269038 25147 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.270098 25147 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.271641 25147 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:18:44.271862 24954 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.272507 25147 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.251505 25148 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240426 13:18:44.277019 25148 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.279157 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.294334 25148 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.300602 25145 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.301064 25145 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.059s	user 0.012s	sys 0.008s
I20240426 13:18:44.301323 25147 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.303562 25145 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.304288 25145 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.304582 25145 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.304756 25148 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.305222 25148 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.028s	user 0.003s	sys 0.011s
I20240426 13:18:44.305212 25145 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.307065 25145 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I20240426 13:18:44.307636 25148 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.308163 25147 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.308326 25148 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.308452 25460 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.308568 25147 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.036s	user 0.009s	sys 0.001s
I20240426 13:18:44.308604 25148 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.309435 25148 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.310611 25145 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.310945 25147 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.311545 25147 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.311789 25147 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.312345 25147 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.314575 25147 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:18:44.315714 25147 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.316468 25148 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.002s
I20240426 13:18:44.317289 25148 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.320544 25147 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.322471 25148 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.322767 25145 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.348591 25147 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.349174 25147 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.012s
I20240426 13:18:44.352315 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.352171 25147 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.353137 25147 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.353478 25147 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.354224 25147 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.356961 25147 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20240426 13:18:44.358034 25147 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.362214 25340 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.362672 25340 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.364814 25147 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.365101 25340 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.366554 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.367282 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.368048 25025 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:44.372480 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.375886 25460 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.377494 25147 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.378094 25147 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.005s
I20240426 13:18:44.380738 25147 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.381400 25147 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.381672 25147 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.382498 25147 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.384907 25147 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20240426 13:18:44.385726 25147 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.386480 25247 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.387248 25460 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.386950 25247 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.391177 25247 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.392395 25147 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.392915 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.393436 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.393947 25104 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.394971 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:18:44.395641 24995 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.396173 25469 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.396507 25469 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.396792 25469 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.397400 25169 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.397769 25169 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.400159 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.400929 25169 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.400882 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.402341 25069 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.407075 25469 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.410466 25025 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:44.413666 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.415006 25169 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.416196 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.417464 24995 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.418514 25469 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.419503 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.423151 25169 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.428143 25145 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.428561 25145 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.118s	user 0.021s	sys 0.014s
I20240426 13:18:44.432202 25469 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.432168 25104 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.432879 25469 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.434264 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.435096 25247 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.435469 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.435706 25247 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:18:44.437043 24995 leader_election.cc:341] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.436859 25247 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.439527 25469 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.451236 25469 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.453503 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.456022 25148 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.456890 25148 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.140s	user 0.023s	sys 0.050s
I20240426 13:18:44.462445 25145 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.463135 25145 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.463384 25145 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.463987 25145 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.454190 25104 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.466854 25460 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.467262 25460 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.469944 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.472188 25460 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.472388 25148 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.473183 25145 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.001s
I20240426 13:18:44.473524 25147 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.475180 25148 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.476022 25148 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.476377 25145 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.476657 25148 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.477314 25388 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.477694 25388 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.478641 25147 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.093s	user 0.041s	sys 0.044s
I20240426 13:18:44.478717 25148 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20240426 13:18:44.479570 25148 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.480199 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.480998 25147 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.482127 25145 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.482710 25147 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.477988 24954 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.483150 25147 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.481493 25388 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:44.482385 24995 leader_election.cc:341] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.486804 25147 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.488163 24854 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:44.491271 25247 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.485638 25148 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.493719 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.494055 25147 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20240426 13:18:44.494534 25247 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.494868 25247 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.494932 25147 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.495175 25247 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.497043 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.497666 25104 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.503594 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.504225 25104 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:44.507988 24921 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.509111 25340 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.509907 25147 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.510025 24996 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.512785 25469 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.511269 25247 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.517969 25247 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.528340 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.513149 25469 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:18:44.531675 24995 leader_election.cc:341] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.536207 25469 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.538102 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.541034 25102 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.541072 25340 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.542028 25247 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.542516 25247 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.546038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.547211 25247 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:44.547359 24995 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.548976 25169 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.549651 25102 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.551628 25169 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.557210 25220 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.594019 25388 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.594535 25388 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.596738 25388 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.598625 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.600145 24995 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.600597 25145 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.601087 25145 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.125s	user 0.013s	sys 0.002s
W20240426 13:18:44.605439 25069 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:44.612424 25476 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20240426 13:18:44.649086 25148 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.649574 25148 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.170s	user 0.008s	sys 0.005s
I20240426 13:18:44.653059 25147 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.653563 25147 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.159s	user 0.029s	sys 0.013s
I20240426 13:18:44.655938 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.656782 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.668563 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.678723 25147 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.679292 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.679510 25147 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.679831 25147 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.680466 25147 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.681375 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.682727 25510 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.682986 25148 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.683569 25509 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.684077 25509 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.684427 25509 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.683810 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:44.683794 25148 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.686348 25148 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.691991 25145 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.692965 25145 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.693291 25145 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.687101 25148 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.698750 25147 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.000s
I20240426 13:18:44.699934 25147 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.696195 25145 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.706163 25148 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.001s
I20240426 13:18:44.706499 25469 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.707794 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:44.708371 25469 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.712450 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.714418 24995 leader_election.cc:341] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.717689 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:44.729259 25169 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:44.702035 24996 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.729920 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:44.731423 25388 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.749344 25169 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.749917 25145 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.149s	user 0.016s	sys 0.002s
I20240426 13:18:44.749879 25169 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.754920 25149 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.755331 25149 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.756888 25149 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.758384 25169 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.758152 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.761008 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.762372 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:44.763312 25169 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.763725 25169 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.764123 25169 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.768628 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.752341 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.790817 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.806080 25145 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.807621 25169 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.817451 25169 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.810760 25510 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:44.816080 25103 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.816162 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.859699 25510 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.889567 25516 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:18:44.891706 25147 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.876242 25027 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.923633 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:18:44.937918 25069 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.939030 25069 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.941195 25519 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.941871 25519 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.944573 25476 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.989790 25476 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.990069 25519 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.985904 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.015215 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:45.017508 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76585.4us
I20240426 13:18:45.015992 25476 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.059618 25145 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:45.067466 25521 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:18:45.089540 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.063863 25509 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.114344 25509 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.091749 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:45.154125 25069 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.143637 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:45.129640 25023 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.178337 25103 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
W20240426 13:18:45.180536 24996 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:45.207374 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45983.7us
I20240426 13:18:45.237300 25147 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.237917 25147 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.538s	user 0.045s	sys 0.053s
I20240426 13:18:45.277374 25145 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
W20240426 13:18:45.268664 24995 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.277824 25145 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.472s	user 0.017s	sys 0.000s
I20240426 13:18:45.277981 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:45.280141 25145 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.280794 25145 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.281035 25145 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.281570 25145 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.282732 25145 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240426 13:18:45.282754 25023 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.256166 25476 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.299422 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:18:45.306221 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58679us
I20240426 13:18:45.305428 25147 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.307574 25147 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.307849 25147 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.298511 25531 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.308450 25147 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.299913 25388 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.321305 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:45.321309 25536 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.269781 24996 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.325333 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.325436 25476 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.306404 25533 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.330307 25147 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.092s	user 0.008s	sys 0.000s
I20240426 13:18:45.333439 25147 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.336850 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:45.337469 25028 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.323012 25536 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.341638 25536 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.349905 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:45.357246 25520 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.303s	user 0.004s	sys 0.000s
I20240426 13:18:45.334803 25533 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.330426 25531 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.360585 24954 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:45.349844 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85093.6us
I20240426 13:18:45.362109 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.362440 25531 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.362920 25025 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:45.360642 25533 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.369011 25544 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.369866 25544 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:45.371042 25544 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.397341 24854 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.401340 25343 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:18:45.401881 25343 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.404888 25028 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.406941 25343 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:45.409471 24995 consensus_peers.cc:485] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:45.412189 25069 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:45.413578 24995 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.418368 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.423952 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.425513 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.426826 25476 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:45.427258 25476 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:45.427618 25476 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.432884 25147 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.435739 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.436437 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.436904 25476 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.437959 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:45.438534 25100 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.438735 25476 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.441107 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:18:45.442044 25525 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.442466 25525 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.442476 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:45.443267 25525 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:45.443753 25069 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.443715 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:45.445371 25069 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.445989 25100 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
W20240426 13:18:45.460340 24995 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:45.471143 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.473078 25147 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.473265 25247 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.473645 25147 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.141s	user 0.009s	sys 0.006s
I20240426 13:18:45.473704 25247 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:45.473960 25538 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:45.474552 25247 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.475997 25147 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.476691 25147 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.476928 25147 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.477502 25147 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.480017 25147 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240426 13:18:45.484851 25147 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.489279 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.490435 25147 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.493149 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:45.493873 25025 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.508806 25340 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.512437 25343 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.513057 25343 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.514863 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.515576 25343 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.522156 25340 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:45.503795 25025 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.530844 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.531678 25476 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.532244 25476 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.533039 25476 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.536098 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.537350 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.551002 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.552122 25343 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.579582 25147 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.579937 25147 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.095s	user 0.037s	sys 0.048s
I20240426 13:18:45.581733 25147 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.582262 25147 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.582463 25147 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.582959 25147 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.584800 25147 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:18:45.585673 25147 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.590191 25147 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.589643 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.591845 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.599486 25147 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.599920 25147 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.004s
I20240426 13:18:45.602066 25147 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.602736 25147 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.602974 25147 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.603569 25147 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.607304 25147 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.002s
I20240426 13:18:45.608093 25147 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.610599 24854 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.613314 25147 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.613054 24854 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.618335 25147 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.618755 25147 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.001s
I20240426 13:18:45.620955 25147 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.621559 25147 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.621827 25147 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.622390 25147 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.624413 25147 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.001s
I20240426 13:18:45.625187 25147 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.630129 25147 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.633138 24954 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.636466 25247 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.639036 24852 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.685688 25147 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.686152 25147 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.061s	user 0.025s	sys 0.032s
I20240426 13:18:45.688338 25147 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.688982 25147 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.689254 25147 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.689792 25147 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.691872 25147 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.001s
I20240426 13:18:45.692725 25147 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.696981 25147 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.701676 25147 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.702126 25147 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.003s
I20240426 13:18:45.704283 25147 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.704733 25147 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.704933 25147 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.705405 25147 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.706854 25147 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:18:45.707468 25147 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.712188 25147 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.723409 25147 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.723860 25147 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.011s
I20240426 13:18:45.726006 25147 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.726652 25147 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.726905 25147 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.727519 25147 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.729601 25147 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20240426 13:18:45.763857 25531 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.764367 25531 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.766121 25531 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.767045 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.767042 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.769428 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.770242 25531 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.807253 24948 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.813416 25528 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.823168 25247 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.826627 24954 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.841818 25531 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.842206 25531 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.843596 25531 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.844229 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.844869 25028 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:45.844631 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.846194 24921 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.847136 25531 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.852551 25531 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.855785 25525 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.865020 24954 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.869930 25533 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.876541 25533 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.882001 25533 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:45.919626 24954 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.926949 25247 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.930404 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:45.942155 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.944239 25343 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.944952 25343 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:45.945730 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.948276 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.949141 25025 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:45.951996 25343 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.953299 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.957528 24921 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.957765 25544 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.958447 25531 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.979741 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:45.980520 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.982230 25531 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.983752 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:45.984112 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:45.984354 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137525984338 us; error 0 us; skew 500 ppm
I20240426 13:18:45.985146 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:45.992588 23796 webserver.cc:460] Webserver started at http://127.23.61.4:46465/ using document root <none> and password file <none>
I20240426 13:18:45.993144 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:45.993309 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:45.993568 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:45.994710 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/instance:
uuid: "1590f80bdade41c6a755b8501c55fe16"
format_stamp: "Formatted at 2024-04-26 13:18:45 on dist-test-slave-jx77"
I20240426 13:18:45.999164 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20240426 13:18:46.002399 25586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:46.003178 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240426 13:18:46.003441 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
uuid: "1590f80bdade41c6a755b8501c55fe16"
format_stamp: "Formatted at 2024-04-26 13:18:45 on dist-test-slave-jx77"
I20240426 13:18:46.003692 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:46.024974 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:18:46.026360 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:46.027802 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:46.030392 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:46.030674 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:46.030978 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:46.031191 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:18:46.113039 25525 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:46.133138 25525 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:46.170192 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.4:40069
I20240426 13:18:46.170292 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.4:40069 every 8 connection(s)
I20240426 13:18:46.205528 23796 auto_leader_rebalancer-test.cc:405] add a tserver: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:46.214516 25651 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:46.214915 25651 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:46.215574 25651 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:46.217203 24851 ts_manager.cc:194] Registered new tserver with Master: 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:18:46.218364 24851 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60836
I20240426 13:18:46.826489 25100 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:46.833271 25025 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.834826 24953 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.835011 25655 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.836373 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.842222 25525 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.843525 24953 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.851163 25025 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:18:46.851903 25069 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e915e90a1a6847899575125ef7e68f2a. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:46.866313 24851 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.876639 24953 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:46.885234 25100 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.885540 25025 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.887223 25531 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.887972 25340 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:46.907922 24920 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4076b67a4fe4b3297158d094e10de08. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:46.920836 25340 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.922333 25028 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.938758 24952 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.013811 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:47.016237 24920 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f0f51d042f24e54b9e3a33e390f5982. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:47.018324 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.020339 25670 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:47.019953 25100 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.022071 25027 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.029992 25672 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.043183 25670 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.044119 25104 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.052625 25027 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.064329 24854 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.075425 25104 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.081804 25031 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.083072 25655 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.088611 24952 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:47.088995 25069 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a28ef3032ef44d2d89c750a8d4892f00. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:47.090783 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.100252 25530 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.103767 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.115046 25031 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.117615 24851 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:47.220291 25651 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:47.327930 25678 ts_tablet_manager.cc:902] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:47.332517 25678 tablet_copy_client.cc:308] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:47.356379 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet e915e90a1a6847899575125ef7e68f2a from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:47.357034 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet e915e90a1a6847899575125ef7e68f2a from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a
I20240426 13:18:47.363461 25115 tablet_copy_source_session.cc:205] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.367094 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.378876 25678 tablet_copy_client.cc:783] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.379418 25678 tablet_copy_client.cc:647] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.383457 25678 tablet_copy_client.cc:523] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.391124 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.408108 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.408793 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.409396 25678 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.018s	user 0.019s	sys 0.000s
I20240426 13:18:47.411641 25678 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.412413 25678 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.413005 25678 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.415668 25678 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240426 13:18:47.417117 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:47.417500 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a on tablet e915e90a1a6847899575125ef7e68f2a with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.420979 25678 ts_tablet_manager.cc:902] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:47.432463 25678 tablet_copy_client.cc:308] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:47.445921 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 8f0f51d042f24e54b9e3a33e390f5982 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:47.446491 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 8f0f51d042f24e54b9e3a33e390f5982 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982
I20240426 13:18:47.453779 24966 tablet_copy_source_session.cc:205] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.456410 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.466979 25678 tablet_copy_client.cc:783] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.467446 25678 tablet_copy_client.cc:647] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.472317 25687 ts_tablet_manager.cc:902] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:47.475312 25678 tablet_copy_client.cc:523] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.482676 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.488080 25687 tablet_copy_client.cc:308] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:47.489789 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet f4076b67a4fe4b3297158d094e10de08 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:47.490343 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet f4076b67a4fe4b3297158d094e10de08 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08
I20240426 13:18:47.497515 24966 tablet_copy_source_session.cc:205] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.500232 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.506209 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.506920 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.507458 25678 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.005s
I20240426 13:18:47.510426 25678 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.511133 25678 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.511603 25678 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.516321 25687 tablet_copy_client.cc:783] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.516829 25687 tablet_copy_client.cc:647] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.519228 25678 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.004s
I20240426 13:18:47.522756 25687 tablet_copy_client.cc:523] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.525921 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:47.526297 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982 on tablet 8f0f51d042f24e54b9e3a33e390f5982 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.533028 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.555169 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.556638 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.557236 25687 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.006s
I20240426 13:18:47.559645 25687 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.560464 25687 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.561110 25687 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.563676 25687 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20240426 13:18:47.565120 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:47.566191 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08 on tablet f4076b67a4fe4b3297158d094e10de08 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.607456 25687 ts_tablet_manager.cc:902] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:47.610935 25687 tablet_copy_client.cc:308] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:47.612406 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet a28ef3032ef44d2d89c750a8d4892f00 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:47.612838 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet a28ef3032ef44d2d89c750a8d4892f00 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00
I20240426 13:18:47.618269 25115 tablet_copy_source_session.cc:205] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.622820 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.636518 25687 tablet_copy_client.cc:783] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.637032 25687 tablet_copy_client.cc:647] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.642630 25687 tablet_copy_client.cc:523] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.650103 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.666124 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.666869 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.667435 25687 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
I20240426 13:18:47.669865 25687 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.670625 25687 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.671721 25687 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.673887 25687 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20240426 13:18:47.675633 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:47.675983 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00 on tablet a28ef3032ef44d2d89c750a8d4892f00 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.705282 25624 raft_consensus.cc:1203] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:47.816872 25624 raft_consensus.cc:1203] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:47.923321 25624 raft_consensus.cc:1203] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:48.021375 25624 raft_consensus.cc:1203] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:48.229637 25672 raft_consensus.cc:1050] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.231947 25672 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.237628 25624 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.238708 25706 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.240613 25104 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.241484 25031 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.242543 25706 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.244088 25672 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.249917 25666 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.256520 25031 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.259442 25624 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.269929 25099 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.275172 24851 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.301064 24948 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.305279 25624 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.305763 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.306443 25666 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.308753 25666 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.312630 25706 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.314966 25705 raft_consensus.cc:1050] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.318531 25099 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.319116 25705 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.321382 25672 raft_consensus.cc:1050] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.323690 25672 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.326419 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f0f51d042f24e54b9e3a33e390f5982 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.326228 25624 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.341295 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.352078 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.353654 25533 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.359977 25031 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.361769 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.365299 24953 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.366580 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.378365 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:48.390395 25624 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.392375 25672 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:48.393044 25533 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.397210 25028 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.399066 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.402194 25028 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.420107 25099 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.421591 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.423954 25723 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:48.427280 25666 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.432864 25027 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.434691 25624 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.441321 24953 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.450798 25723 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.451895 25723 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.453533 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.457679 25624 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.476174 25723 ts_tablet_manager.cc:1875] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.486044 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:48.487012 25104 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.502538 25723 ts_tablet_manager.cc:1888] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.503041 25723 log.cc:1196] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/8f0f51d042f24e54b9e3a33e390f5982
I20240426 13:18:48.505415 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 8f0f51d042f24e54b9e3a33e390f5982 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.508299 25104 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.512339 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.513257 25031 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.513634 25705 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.514560 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.521955 25530 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.526067 24953 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.530419 25031 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.532763 25623 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.533866 25533 raft_consensus.cc:1050] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.535830 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.536146 25533 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.537492 25028 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.541508 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.546309 25622 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.552376 25666 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.565212 24854 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.567903 25622 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.591521 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4076b67a4fe4b3297158d094e10de08 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.603896 25027 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.608283 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.630558 25031 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.632150 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.634680 24953 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.636488 25705 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.638643 25622 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.640993 25722 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.654731 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet e915e90a1a6847899575125ef7e68f2a with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:48.658466 25734 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:48.670106 25734 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.670925 25734 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.674072 25734 ts_tablet_manager.cc:1875] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.706372 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e915e90a1a6847899575125ef7e68f2a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.743126 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.754417 25533 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.765748 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet f4076b67a4fe4b3297158d094e10de08 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:48.771764 25027 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:18:48.805292 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48469.7us
I20240426 13:18:48.813553 25622 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.820850 25737 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:48.821174 24851 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.827250 25734 ts_tablet_manager.cc:1888] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.827991 25734 log.cc:1196] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/e915e90a1a6847899575125ef7e68f2a
I20240426 13:18:48.830166 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet e915e90a1a6847899575125ef7e68f2a (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.857878 25737 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.863714 25737 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.873905 25104 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.878955 25737 ts_tablet_manager.cc:1875] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.880198 25622 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.880179 24948 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.882046 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.889048 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.891849 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.893781 25737 ts_tablet_manager.cc:1888] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.894222 25737 log.cc:1196] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/f4076b67a4fe4b3297158d094e10de08
I20240426 13:18:48.895793 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet f4076b67a4fe4b3297158d094e10de08 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.890522 25530 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.899255 25622 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.908447 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a28ef3032ef44d2d89c750a8d4892f00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.936909 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.954304 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:48.955641 25723 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:48.973716 25723 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.974592 25723 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.979606 25723 ts_tablet_manager.cc:1875] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.992491 25723 ts_tablet_manager.cc:1888] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.992846 25723 log.cc:1196] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/a28ef3032ef44d2d89c750a8d4892f00
I20240426 13:18:48.994138 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet a28ef3032ef44d2d89c750a8d4892f00 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:50.006062 25104 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.013796 24948 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.014156 25027 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:50.014070 25069 consensus_peers.cc:485] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4680c139c46f44b49096bfd65f033a32. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.015159 25739 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.015949 25533 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.024395 24948 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.030057 25027 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.022917 25533 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.039239 24854 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.074087 24948 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:50.092530 24920 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 786abf3f43724ad5b959e497fc776d1d. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.092675 25104 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.094115 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.097270 25743 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.099260 25666 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:50.114991 25744 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.129842 25105 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.135630 24953 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:50.150375 24920 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3a669ffe77a4ad29604f4ecd7c2644a. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.147704 25031 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.146884 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.157034 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.158200 25105 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.159240 25666 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.160166 25743 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.175819 25743 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.186203 25105 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.193562 25031 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.197342 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.204331 25028 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.222267 25105 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.222569 24953 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.224629 25751 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.225574 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:50.233036 24995 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a2ec72789da4a7383ca5ce0f90a4563. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.234961 25751 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.236155 24948 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.238086 25105 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.252436 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.574813 25762 ts_tablet_manager.cc:902] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:50.608137 25762 tablet_copy_client.cc:308] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:50.609663 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 4680c139c46f44b49096bfd65f033a32 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:50.610153 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 4680c139c46f44b49096bfd65f033a32 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32
I20240426 13:18:50.616380 25115 tablet_copy_source_session.cc:205] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.619112 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.629402 25767 ts_tablet_manager.cc:902] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:50.639420 25762 tablet_copy_client.cc:783] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.640480 25762 tablet_copy_client.cc:647] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.644645 25767 tablet_copy_client.cc:308] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:50.647989 25762 tablet_copy_client.cc:523] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.651793 25773 ts_tablet_manager.cc:902] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:50.658587 25773 tablet_copy_client.cc:308] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:50.662590 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 786abf3f43724ad5b959e497fc776d1d from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:50.663107 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 786abf3f43724ad5b959e497fc776d1d from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d
I20240426 13:18:50.663457 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.665414 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 8a2ec72789da4a7383ca5ce0f90a4563 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:50.665997 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 8a2ec72789da4a7383ca5ce0f90a4563 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563
I20240426 13:18:50.674685 24966 tablet_copy_source_session.cc:205] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.677501 25041 tablet_copy_source_session.cc:205] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.679965 25767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.681298 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.714887 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.716423 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.717061 25762 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.054s	user 0.028s	sys 0.011s
I20240426 13:18:50.719095 25773 tablet_copy_client.cc:783] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.720983 25779 ts_tablet_manager.cc:902] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:50.719741 25762 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.723388 25762 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.723858 25767 tablet_copy_client.cc:783] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.724459 25767 tablet_copy_client.cc:647] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.724964 25773 tablet_copy_client.cc:647] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.724066 25762 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.729677 25767 tablet_copy_client.cc:523] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.732482 25779 tablet_copy_client.cc:308] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:50.736989 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet d3a669ffe77a4ad29604f4ecd7c2644a from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:50.737527 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet d3a669ffe77a4ad29604f4ecd7c2644a from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a
I20240426 13:18:50.737974 25762 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.021s	user 0.010s	sys 0.001s
I20240426 13:18:50.741219 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:50.741624 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32 on tablet 4680c139c46f44b49096bfd65f033a32 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.746167 24965 tablet_copy_source_session.cc:205] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.750715 25773 tablet_copy_client.cc:523] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.755739 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.757926 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.762075 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.782711 25779 tablet_copy_client.cc:783] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.783242 25779 tablet_copy_client.cc:647] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.786840 25779 tablet_copy_client.cc:523] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.793212 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.794307 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.794966 25773 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.033s	user 0.016s	sys 0.007s
I20240426 13:18:50.798077 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.798283 25773 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.798990 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.799268 25773 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.799748 25767 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.044s	user 0.024s	sys 0.007s
I20240426 13:18:50.799953 25773 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.802019 25773 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20240426 13:18:50.802492 25767 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.803373 25767 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.803800 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:50.804229 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d on tablet 786abf3f43724ad5b959e497fc776d1d with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.803951 25767 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.805538 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.806308 25767 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.002s
I20240426 13:18:50.807772 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:50.808197 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563 on tablet 8a2ec72789da4a7383ca5ce0f90a4563 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.831256 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.832229 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.832898 25779 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.004s
I20240426 13:18:50.835075 25779 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.835621 25779 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.836069 25779 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.838259 25779 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.003s
I20240426 13:18:50.839499 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:50.839851 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a on tablet d3a669ffe77a4ad29604f4ecd7c2644a with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:51.069008 25622 raft_consensus.cc:1203] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:51.199012 25622 raft_consensus.cc:1203] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:51.226763 25622 raft_consensus.cc:1203] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:51.240890 25622 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:51.592974 25789 raft_consensus.cc:1050] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.595327 25789 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.600955 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.601598 25622 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.602257 25789 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 13:18:51.608490 24948 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.608378 25751 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.611637 25792 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:51.635389 25751 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.637683 24948 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.640980 25622 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.652123 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.660604 25790 raft_consensus.cc:1050] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.663007 25790 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.659673 25099 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.680591 25622 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.682744 25790 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.688356 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.688584 25099 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.691300 25790 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:51.692549 25744 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.710956 25801 raft_consensus.cc:1050] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.713357 25801 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.724493 25027 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.726487 24948 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.728202 25800 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.728871 25622 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.730080 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.732156 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.736451 25744 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.742054 25027 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.739856 25099 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.796662 25028 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.800803 25622 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.804611 25789 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.807399 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.808974 25792 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.814325 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.816243 25104 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.817382 25739 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.822872 25623 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.828912 25624 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.833739 25622 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.814905 25247 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.852490 24854 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:51.862551 25744 raft_consensus.cc:1050] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.848667 25031 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.865151 25744 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.873240 24953 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.873538 24854 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.879063 24948 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.880563 25623 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.867735 25104 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.907863 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.912791 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a2ec72789da4a7383ca5ce0f90a4563 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:51.917819 25811 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.921047 25623 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.922435 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.932397 25099 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.933410 25800 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.938414 25622 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.939469 25028 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.939903 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:51.939554 25743 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.941525 25790 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.942823 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.944080 25790 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.949266 25815 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:51.950541 25815 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:51.952419 25815 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:51.953454 25785 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.954130 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.958492 25815 ts_tablet_manager.cc:1875] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:51.960337 25790 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.967072 25031 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.971464 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4680c139c46f44b49096bfd65f033a32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:51.973887 24851 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.968163 25623 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.985059 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 4680c139c46f44b49096bfd65f033a32 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:51.990953 25743 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.996795 25743 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.998296 25620 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.005308 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.020814 25099 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.013475 25815 ts_tablet_manager.cc:1888] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.024956 25815 log.cc:1196] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/8a2ec72789da4a7383ca5ce0f90a4563
I20240426 13:18:52.026788 25815 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:52.027670 25815 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.028321 25815 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.029111 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 8a2ec72789da4a7383ca5ce0f90a4563 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.030129 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:52.032456 25023 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.033852 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 786abf3f43724ad5b959e497fc776d1d with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:52.050668 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 786abf3f43724ad5b959e497fc776d1d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:52.063310 25815 ts_tablet_manager.cc:1875] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.066633 25820 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:52.068290 25820 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.069011 25820 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.077778 25820 ts_tablet_manager.cc:1875] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.081666 25815 ts_tablet_manager.cc:1888] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.082134 25815 log.cc:1196] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/4680c139c46f44b49096bfd65f033a32
I20240426 13:18:52.084797 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 4680c139c46f44b49096bfd65f033a32 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.084903 25028 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.097409 25820 ts_tablet_manager.cc:1888] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.097923 25820 log.cc:1196] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/786abf3f43724ad5b959e497fc776d1d
I20240426 13:18:52.100286 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 786abf3f43724ad5b959e497fc776d1d (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.135727 24951 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:52.141619 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:52.142179 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:52.142972 25818 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:52.143923 25810 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:52.149430 25818 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.153329 25620 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.151984 25028 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.158563 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3a669ffe77a4ad29604f4ecd7c2644a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:52.168256 24852 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.176790 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:52.177583 25820 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:52.178306 25820 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.178740 25820 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.181722 25820 ts_tablet_manager.cc:1875] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.193675 25820 ts_tablet_manager.cc:1888] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.194017 25820 log.cc:1196] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/d3a669ffe77a4ad29604f4ecd7c2644a
I20240426 13:18:52.195362 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet d3a669ffe77a4ad29604f4ecd7c2644a (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:53.225579 24953 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:53.231776 24920 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f49912f064d045d8a1d8be2c08b97d44. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.232656 25028 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.232656 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.234134 25818 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.235033 25810 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.242494 25810 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.244237 25104 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.260104 24951 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.262152 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:53.275815 25104 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:53.276543 24920 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2212db2dbcf486888aefb7dd0174287. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.278766 25824 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.280958 25024 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.283305 25824 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:53.296258 25028 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.298621 25818 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.314462 25104 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.314219 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:18:53.322115 25069 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3014c46944bc4cbb82624ac909221634. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.325395 25099 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.326581 24951 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.327839 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.330459 25028 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.333269 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.345095 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:53.363379 25801 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.365135 24951 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.374058 25028 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.378552 25099 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.389781 24853 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:53.399066 25069 consensus_peers.cc:485] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4010c37c00d94e78b5e3260ad78e7904. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.401839 24951 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.402364 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.403501 25801 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.405297 25800 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.413429 25801 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.414808 24951 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.433481 24851 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.445396 25028 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.727800 25842 ts_tablet_manager.cc:902] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:53.731966 25842 tablet_copy_client.cc:308] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:53.734016 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet a2212db2dbcf486888aefb7dd0174287 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:53.734459 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet a2212db2dbcf486888aefb7dd0174287 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287
I20240426 13:18:53.740015 24966 tablet_copy_source_session.cc:205] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.743647 25842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.776082 25842 tablet_copy_client.cc:783] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.776814 25842 tablet_copy_client.cc:647] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.780441 25842 tablet_copy_client.cc:523] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.787982 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.811697 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:53.812543 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:53.813133 25842 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.009s
I20240426 13:18:53.815642 25842 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.816593 25842 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:53.817205 25842 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.820102 25842 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20240426 13:18:53.822152 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:53.822556 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287 on tablet a2212db2dbcf486888aefb7dd0174287 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:53.838300 25842 ts_tablet_manager.cc:902] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:53.844012 25842 tablet_copy_client.cc:308] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:53.845317 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet f49912f064d045d8a1d8be2c08b97d44 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:53.845752 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet f49912f064d045d8a1d8be2c08b97d44 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44
I20240426 13:18:53.853088 24966 tablet_copy_source_session.cc:205] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.855602 25842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.859638 25853 ts_tablet_manager.cc:902] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:53.867772 25842 tablet_copy_client.cc:783] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.868428 25842 tablet_copy_client.cc:647] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.874514 25842 tablet_copy_client.cc:523] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.882189 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.905387 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:53.906479 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:53.907130 25842 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.004s
I20240426 13:18:53.907829 25853 tablet_copy_client.cc:308] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:53.909432 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 4010c37c00d94e78b5e3260ad78e7904 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:53.909919 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 4010c37c00d94e78b5e3260ad78e7904 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904
I20240426 13:18:53.910305 25842 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.911141 25842 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:53.911744 25842 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.916289 25115 tablet_copy_source_session.cc:205] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.920512 25842 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.004s
I20240426 13:18:53.923949 25859 ts_tablet_manager.cc:902] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:53.930099 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:53.930716 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44 on tablet f49912f064d045d8a1d8be2c08b97d44 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:53.935576 25853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.944118 25859 tablet_copy_client.cc:308] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:53.945808 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 3014c46944bc4cbb82624ac909221634 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:53.946328 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 3014c46944bc4cbb82624ac909221634 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634
I20240426 13:18:53.954093 25115 tablet_copy_source_session.cc:205] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.954476 25853 tablet_copy_client.cc:783] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.955070 25853 tablet_copy_client.cc:647] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.958595 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.959684 25853 tablet_copy_client.cc:523] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.973110 25859 tablet_copy_client.cc:783] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.973718 25859 tablet_copy_client.cc:647] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.976321 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.992342 25859 tablet_copy_client.cc:523] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:54.013170 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:54.014508 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:54.015527 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:54.016276 25853 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.040s	user 0.016s	sys 0.021s
I20240426 13:18:54.019222 25853 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.020246 25853 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:54.020962 25853 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.028759 25853 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.004s
I20240426 13:18:54.031286 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:54.031826 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904 on tablet 4010c37c00d94e78b5e3260ad78e7904 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:54.045656 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:54.046636 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:54.047314 25859 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.034s	user 0.024s	sys 0.008s
I20240426 13:18:54.049947 25859 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.050743 25859 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:54.051349 25859 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.053966 25859 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.003s
I20240426 13:18:54.055462 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:54.055840 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634 on tablet 3014c46944bc4cbb82624ac909221634 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:54.143039 25623 raft_consensus.cc:1203] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:54.223541 25623 raft_consensus.cc:1203] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:54.388173 25623 raft_consensus.cc:1203] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:54.447331 25623 raft_consensus.cc:1203] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:54.626962 25866 raft_consensus.cc:1050] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.629279 25866 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.637151 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.637681 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.638707 25824 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.639130 25024 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.640287 25866 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.641127 25824 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.649780 25831 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.652242 25623 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.663743 25024 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.667017 24852 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.689015 25099 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.710702 25866 raft_consensus.cc:1050] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.714866 25866 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.721395 25623 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.722862 25104 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.724560 25866 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.726260 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.726392 25831 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.730122 25831 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.739257 24951 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.745576 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.746022 25099 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.746927 25871 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.747920 25099 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.751405 25872 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.742640 25831 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.753758 25873 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.747700 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.756342 25623 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.756563 25105 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.768644 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f49912f064d045d8a1d8be2c08b97d44 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:54.771919 25623 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.775703 24851 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.777936 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.806648 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet f49912f064d045d8a1d8be2c08b97d44 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:54.813918 25883 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:54.813331 24951 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.815016 25883 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:54.820977 25883 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:54.824467 25883 ts_tablet_manager.cc:1875] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:54.830653 25623 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.833101 25866 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.834162 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.835690 25872 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.836591 25800 raft_consensus.cc:1050] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.838806 25800 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.843976 25866 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.848352 25623 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.854110 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.857388 25883 ts_tablet_manager.cc:1888] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:54.857810 25883 log.cc:1196] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/f49912f064d045d8a1d8be2c08b97d44
I20240426 13:18:54.859179 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet f49912f064d045d8a1d8be2c08b97d44 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:54.861816 25620 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.863267 25024 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.863643 24954 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.865172 25801 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.866063 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.868155 25800 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.871729 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2212db2dbcf486888aefb7dd0174287 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:54.878091 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.892760 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet a2212db2dbcf486888aefb7dd0174287 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:54.896920 25887 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:54.902024 25887 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:54.902549 25887 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:54.906528 25887 ts_tablet_manager.cc:1875] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:54.918270 25800 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.925550 25623 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.928622 25887 ts_tablet_manager.cc:1888] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:54.929145 25887 log.cc:1196] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/a2212db2dbcf486888aefb7dd0174287
I20240426 13:18:54.930431 25024 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.930846 24954 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.937450 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet a2212db2dbcf486888aefb7dd0174287 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:54.946561 24852 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:54.963316 25099 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.969354 24954 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.972153 25623 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.971413 25800 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:54.973387 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.980775 25801 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.983352 25623 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.987736 24954 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.003703 25825 raft_consensus.cc:1050] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:55.003402 24853 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.006012 25825 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:55.008661 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3014c46944bc4cbb82624ac909221634 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:55.018595 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3014c46944bc4cbb82624ac909221634 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:55.019615 25883 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:55.022804 25623 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.023370 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.024838 25885 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.027278 25885 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.032789 24953 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.034003 25825 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.047252 25890 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.052565 25028 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.057929 25883 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:55.058701 25883 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:55.059232 25622 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.061921 25883 ts_tablet_manager.cc:1875] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:55.067788 24954 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.074064 24853 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.095669 25883 ts_tablet_manager.cc:1888] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:55.096220 25883 log.cc:1196] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/3014c46944bc4cbb82624ac909221634
I20240426 13:18:55.097862 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 3014c46944bc4cbb82624ac909221634 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:55.107892 25105 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:55.112452 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:55.114296 25622 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:55.114290 25890 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:55.117297 25890 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:55.121201 25801 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.122189 25024 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.127765 25622 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.130074 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4010c37c00d94e78b5e3260ad78e7904 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:55.140839 24853 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.154361 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:55.164561 25895 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:55.167372 25895 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:55.168115 25895 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:55.170822 25895 ts_tablet_manager.cc:1875] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:55.182121 25895 ts_tablet_manager.cc:1888] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:55.182444 25895 log.cc:1196] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/4010c37c00d94e78b5e3260ad78e7904
I20240426 13:18:55.183744 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 4010c37c00d94e78b5e3260ad78e7904 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:56.196419 25024 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.202286 25105 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:56.202785 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59632e528afa44ca98a626355e1e4660. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.203804 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.204438 24954 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.206842 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.213286 25841 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.217233 25105 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.219576 24953 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.230674 24852 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.234701 25024 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:56.242067 24995 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fbdcc6d88d14934b64bc2a29382519f. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.242452 24954 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.244304 25105 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.244347 25751 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.245747 25900 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.256537 25841 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.257953 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.259173 24954 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.276935 25099 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.284691 25024 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.285636 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.287264 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:56.288563 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f413a75d3c249dbb47c1ec61978c2ea. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.287873 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.289350 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.306808 25825 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.309729 24954 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.313838 25028 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.326049 24953 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:56.333786 24920 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be16e3d2b66a44c5ac4a072b891540a2. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.335512 25099 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.335413 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.336259 25028 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.336956 25824 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.337874 25866 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.355154 25866 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.359668 25099 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.364403 25028 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.373754 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:56.624153 25916 ts_tablet_manager.cc:902] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:56.630928 25916 tablet_copy_client.cc:308] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:56.632521 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 59632e528afa44ca98a626355e1e4660 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:56.633020 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 59632e528afa44ca98a626355e1e4660 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660
I20240426 13:18:56.639328 25041 tablet_copy_source_session.cc:205] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.644604 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.658077 25916 tablet_copy_client.cc:783] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.658564 25916 tablet_copy_client.cc:647] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.661454 25916 tablet_copy_client.cc:523] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.668095 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.692282 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.693157 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.693761 25916 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.026s	user 0.022s	sys 0.000s
I20240426 13:18:56.696115 25916 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.696686 25916 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.697307 25916 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.699898 25916 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.004s
I20240426 13:18:56.701231 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:56.701572 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660 on tablet 59632e528afa44ca98a626355e1e4660 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.820650 25916 ts_tablet_manager.cc:902] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:56.824151 25916 tablet_copy_client.cc:308] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:56.825520 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 0fbdcc6d88d14934b64bc2a29382519f from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:56.826016 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 0fbdcc6d88d14934b64bc2a29382519f from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f
I20240426 13:18:56.832777 25041 tablet_copy_source_session.cc:205] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.835893 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.849298 25916 tablet_copy_client.cc:783] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.849869 25916 tablet_copy_client.cc:647] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.853013 25916 tablet_copy_client.cc:523] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.860098 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.869571 25927 ts_tablet_manager.cc:902] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:56.886387 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.887337 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.888073 25916 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.028s	user 0.024s	sys 0.001s
I20240426 13:18:56.890434 25916 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.891256 25916 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.891929 25916 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.894686 25927 tablet_copy_client.cc:308] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:56.894724 25916 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240426 13:18:56.899391 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 7f413a75d3c249dbb47c1ec61978c2ea from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:56.899883 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 7f413a75d3c249dbb47c1ec61978c2ea from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea
I20240426 13:18:56.900571 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:56.901022 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f on tablet 0fbdcc6d88d14934b64bc2a29382519f with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.906963 25114 tablet_copy_source_session.cc:205] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.910073 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.913739 25916 ts_tablet_manager.cc:902] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:56.922430 25927 tablet_copy_client.cc:783] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.922914 25927 tablet_copy_client.cc:647] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.926467 25927 tablet_copy_client.cc:523] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.928491 25916 tablet_copy_client.cc:308] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:56.931247 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet be16e3d2b66a44c5ac4a072b891540a2 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:56.931849 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet be16e3d2b66a44c5ac4a072b891540a2 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2
I20240426 13:18:56.940616 24966 tablet_copy_source_session.cc:205] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.948838 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.959020 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.962914 25916 tablet_copy_client.cc:783] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.963443 25916 tablet_copy_client.cc:647] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.968056 25916 tablet_copy_client.cc:523] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.975446 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.977679 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.978585 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.979209 25927 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.020s	user 0.018s	sys 0.000s
I20240426 13:18:56.981926 25927 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.982748 25927 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.983341 25927 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.986285 25927 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20240426 13:18:56.987740 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:56.988152 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea on tablet 7f413a75d3c249dbb47c1ec61978c2ea with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.999377 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:57.000203 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:57.001224 25916 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.026s	user 0.020s	sys 0.001s
I20240426 13:18:57.003489 25916 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:57.004298 25916 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:57.004819 25916 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:57.006474 25916 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240426 13:18:57.007714 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:57.008260 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2 on tablet be16e3d2b66a44c5ac4a072b891540a2 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:57.151690 25622 raft_consensus.cc:1203] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:57.229954 25622 raft_consensus.cc:1203] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:57.342352 25622 raft_consensus.cc:1203] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:57.484723 25622 raft_consensus.cc:1203] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:18:57.615234 25939 raft_consensus.cc:1050] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.617318 25939 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.622464 25622 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.622838 25099 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.623359 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.623736 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.624914 25940 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.625731 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.644732 25751 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.646504 25099 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.656915 25622 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.657617 24954 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.668888 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.700315 25031 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.706569 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.707881 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.717337 25940 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.718456 24954 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.721096 25622 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.722338 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.727564 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59632e528afa44ca98a626355e1e4660 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:57.729221 25622 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.737557 24851 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:57.753280 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 59632e528afa44ca98a626355e1e4660 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:57.767735 25951 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:57.768839 25951 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:57.769374 25951 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:57.771883 25951 ts_tablet_manager.cc:1875] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:57.793289 25951 ts_tablet_manager.cc:1888] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:57.793687 25951 log.cc:1196] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/59632e528afa44ca98a626355e1e4660
I20240426 13:18:57.794857 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 59632e528afa44ca98a626355e1e4660 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:57.838855 25940 raft_consensus.cc:1050] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.840932 25940 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.846885 25622 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.847671 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.849395 24954 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.849951 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:57.850947 25940 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.863031 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.867779 25939 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.868891 25622 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.876675 24953 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.878646 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.881634 24854 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.898689 25959 raft_consensus.cc:1050] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.910522 25024 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.908663 25959 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.915374 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.916553 25955 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.919689 25622 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.925176 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.925611 25105 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.925060 25940 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.923885 25622 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.926489 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.927536 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.929574 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.935375 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fbdcc6d88d14934b64bc2a29382519f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:57.936209 25622 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.941623 25031 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.944528 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.947400 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:57.960638 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:57.987248 25824 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.989207 25099 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.005076 25622 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.006508 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.020666 25964 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:58.020277 25028 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.022183 25964 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.022907 25964 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.026538 25964 ts_tablet_manager.cc:1875] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.064466 24954 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.073145 25622 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.074669 25824 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.079007 25964 ts_tablet_manager.cc:1888] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.079517 25964 log.cc:1196] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/0fbdcc6d88d14934b64bc2a29382519f
I20240426 13:18:58.081748 25622 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.080405 25959 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.083606 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.086620 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 0fbdcc6d88d14934b64bc2a29382519f (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:58.087888 25942 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.094753 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be16e3d2b66a44c5ac4a072b891540a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:58.095463 25801 raft_consensus.cc:1050] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:58.097676 25801 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.098747 25104 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.109737 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.109627 24852 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.111053 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.111142 25028 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.112442 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.116168 25622 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.117586 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.128346 25825 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.130404 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:58.132359 25969 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:58.133339 25969 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.134076 25969 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.135082 25028 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.138366 25969 ts_tablet_manager.cc:1875] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.142772 25622 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.159469 25969 ts_tablet_manager.cc:1888] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.159924 25969 log.cc:1196] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/be16e3d2b66a44c5ac4a072b891540a2
I20240426 13:18:58.160482 24953 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.165184 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet be16e3d2b66a44c5ac4a072b891540a2 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:58.169804 24851 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.189563 25105 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.194751 25622 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.194986 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.195974 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.196753 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.201848 25801 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.206001 25622 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.209954 24954 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.211107 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f413a75d3c249dbb47c1ec61978c2ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:58.232669 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.245393 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:58.246258 25969 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:58.247165 25969 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.247748 25969 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.250881 25969 ts_tablet_manager.cc:1875] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.265786 25969 ts_tablet_manager.cc:1888] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.266142 25969 log.cc:1196] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/7f413a75d3c249dbb47c1ec61978c2ea
I20240426 13:18:58.267271 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 7f413a75d3c249dbb47c1ec61978c2ea (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:59.301422 25024 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:59.309525 24995 consensus_peers.cc:485] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d066b73b2e464d80b7b667e05602d879. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.309827 24954 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.310523 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.311933 25971 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.313211 25940 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.322439 24951 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.321236 25971 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.326931 25104 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.344908 24852 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.345774 25028 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:59.362449 24995 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edc419d5057b449c8d0888a41017ce2e. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.362493 24951 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.364362 25978 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.366495 25104 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.375761 25751 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.376580 24951 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.376644 25971 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.386862 25099 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.396835 25104 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.397182 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.411660 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.413062 25825 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:59.418092 25069 consensus_peers.cc:485] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a876ca7b6e4082b58359403a47730d. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.420697 24951 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.422183 25801 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.437270 25825 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.438686 25028 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.440394 24951 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.455523 25104 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.456389 24853 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:59.468987 25028 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:59.470305 25069 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0238fe3803ea4654ae30109aa503e3a3. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.470672 25970 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.471460 24951 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.473116 25970 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.481915 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.490777 24954 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.491387 25028 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.508062 24854 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.770419 25993 ts_tablet_manager.cc:902] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:59.775177 25993 tablet_copy_client.cc:308] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:59.776942 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet d066b73b2e464d80b7b667e05602d879 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:59.777524 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet d066b73b2e464d80b7b667e05602d879 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879
I20240426 13:18:59.784606 25041 tablet_copy_source_session.cc:205] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:59.787808 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:59.802915 25993 tablet_copy_client.cc:783] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:59.803588 25993 tablet_copy_client.cc:647] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:59.808915 25993 tablet_copy_client.cc:523] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:59.819212 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:59.840114 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:59.841004 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:59.841676 25993 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.005s
I20240426 13:18:59.844214 25993 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.845171 25993 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:59.845824 25993 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.849867 25993 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.006s
I20240426 13:18:59.851168 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:59.851574 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879 on tablet d066b73b2e464d80b7b667e05602d879 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:59.911662 25993 ts_tablet_manager.cc:902] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:59.925277 25993 tablet_copy_client.cc:308] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:59.927604 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 0238fe3803ea4654ae30109aa503e3a3 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:59.928133 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 0238fe3803ea4654ae30109aa503e3a3 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3
I20240426 13:18:59.936527 25114 tablet_copy_source_session.cc:205] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:59.939178 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:59.955466 25993 tablet_copy_client.cc:783] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:59.957197 25993 tablet_copy_client.cc:647] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:59.957490 26004 ts_tablet_manager.cc:902] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:59.962553 25993 tablet_copy_client.cc:523] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:59.971526 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:59.977526 26007 ts_tablet_manager.cc:902] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:00.002734 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.003829 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.004698 25993 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.009s
I20240426 13:19:00.012919 26004 tablet_copy_client.cc:308] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:00.074890 25993 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.075839 25993 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.076503 25993 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.083178 25993 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.078s	user 0.040s	sys 0.034s
I20240426 13:19:00.101538 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 99a876ca7b6e4082b58359403a47730d from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:00.102150 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 99a876ca7b6e4082b58359403a47730d from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d
I20240426 13:19:00.109226 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:00.109639 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3 on tablet 0238fe3803ea4654ae30109aa503e3a3 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.114236 26007 tablet_copy_client.cc:308] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:00.116364 25114 tablet_copy_source_session.cc:205] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:00.118003 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet edc419d5057b449c8d0888a41017ce2e from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:00.118533 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet edc419d5057b449c8d0888a41017ce2e from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e
I20240426 13:19:00.120348 26004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:00.133128 25041 tablet_copy_source_session.cc:205] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:00.140252 26007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:00.143402 26004 tablet_copy_client.cc:783] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:00.143981 26004 tablet_copy_client.cc:647] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:00.148052 26004 tablet_copy_client.cc:523] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:00.153106 26007 tablet_copy_client.cc:783] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:00.153693 26007 tablet_copy_client.cc:647] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:00.156414 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:00.157876 26007 tablet_copy_client.cc:523] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:00.171816 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:00.177421 25623 raft_consensus.cc:1203] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:00.214015 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.215214 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.216128 26004 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.060s	user 0.032s	sys 0.013s
I20240426 13:19:00.220667 26004 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.221599 26004 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.222246 26004 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.227188 26004 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20240426 13:19:00.227279 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.228178 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.228725 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:00.228893 26007 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.057s	user 0.019s	sys 0.019s
I20240426 13:19:00.229126 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d on tablet 99a876ca7b6e4082b58359403a47730d with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.231523 26007 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.232365 26007 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.232934 26007 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.235967 26007 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20240426 13:19:00.241431 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:00.241825 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e on tablet edc419d5057b449c8d0888a41017ce2e with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.333727 25623 raft_consensus.cc:1203] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:00.394454 25623 raft_consensus.cc:1203] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:00.508931 25623 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:00.686053 26016 raft_consensus.cc:1050] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.688557 26016 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.695690 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.696221 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.696442 24951 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.698252 25751 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.698931 25971 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.699810 26016 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.723372 26016 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.724686 25623 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.732304 25099 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.734202 24951 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.737568 24854 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.767899 25028 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.773276 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.774092 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.776093 25751 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.776795 26016 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.785106 26016 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.795241 25620 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.796653 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d066b73b2e464d80b7b667e05602d879 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:00.801748 25986 raft_consensus.cc:1050] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.804095 25986 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.813707 25623 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.813014 24853 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:00.815106 25100 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.816970 25028 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.818151 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.827376 25825 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.834836 24954 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.836184 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:00.840636 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet d066b73b2e464d80b7b667e05602d879 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:00.848726 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.851570 26033 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:00.852692 26033 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:00.854810 25031 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.864002 26033 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:00.865681 25620 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.867722 24954 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.875187 26033 ts_tablet_manager.cc:1875] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:00.879092 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.889506 26033 ts_tablet_manager.cc:1888] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:00.889925 26033 log.cc:1196] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/d066b73b2e464d80b7b667e05602d879
I20240426 13:19:00.892160 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet d066b73b2e464d80b7b667e05602d879 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:00.912358 25100 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.919200 25623 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.920692 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.921202 24951 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.922423 25986 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.926403 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.928972 25623 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.935842 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0238fe3803ea4654ae30109aa503e3a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:00.937171 24951 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.939442 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.950425 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:00.960610 26035 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:00.961799 26035 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:00.962443 26035 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:00.969537 26035 ts_tablet_manager.cc:1875] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:00.981601 26035 ts_tablet_manager.cc:1888] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:00.981959 26035 log.cc:1196] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/0238fe3803ea4654ae30109aa503e3a3
I20240426 13:19:00.983361 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 0238fe3803ea4654ae30109aa503e3a3 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:00.995960 26016 raft_consensus.cc:1050] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.998164 26016 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.003784 25100 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.004099 24951 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.004230 25623 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.005329 25971 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.006354 26016 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.007784 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.027873 25971 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.029115 25099 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.036195 25623 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.042851 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:01.046979 24951 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.057014 25028 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.069000 25623 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.070199 24954 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.070216 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:01.073226 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.078960 26016 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.082155 25620 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.093313 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edc419d5057b449c8d0888a41017ce2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:01.098066 24954 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.102898 24851 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:01.117213 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet edc419d5057b449c8d0888a41017ce2e with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:01.122251 26041 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:01.134954 25825 raft_consensus.cc:1050] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:01.136847 25825 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.141232 26041 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:01.141939 26041 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:01.142513 25620 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.144532 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:01.151207 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.152546 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.153234 26041 ts_tablet_manager.cc:1875] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:01.160465 24954 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.163919 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.176910 25825 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.178527 24954 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.184266 25620 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.186139 26041 ts_tablet_manager.cc:1888] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:01.186646 26041 log.cc:1196] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/edc419d5057b449c8d0888a41017ce2e
I20240426 13:19:01.188800 25028 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.198423 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.207475 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet edc419d5057b449c8d0888a41017ce2e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:01.219924 25099 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.224910 25620 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.225299 25023 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.226051 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.229921 26046 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.236726 26046 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.239255 25620 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.242074 25023 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.245644 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99a876ca7b6e4082b58359403a47730d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:01.251893 24851 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.261547 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a876ca7b6e4082b58359403a47730d with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:01.262354 26033 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:01.263240 26033 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:01.263768 26033 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:01.266338 26033 ts_tablet_manager.cc:1875] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:01.280411 26033 ts_tablet_manager.cc:1888] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:01.280718 26033 log.cc:1196] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/99a876ca7b6e4082b58359403a47730d
I20240426 13:19:01.281839 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 99a876ca7b6e4082b58359403a47730d (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:02.309391 24954 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.315255 25028 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.315665 24920 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 256415e42b6c444387115ab766a5a16c. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.316437 26048 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.316906 25099 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.318037 25824 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.323326 26048 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.324843 25028 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.325428 25099 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.339781 25099 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.341650 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.347852 24954 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.348134 25069 consensus_peers.cc:485] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c896b58b7766401eac0a7f69574827f1. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.349961 25825 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.355594 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.357256 26018 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.377382 26053 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.378473 25028 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.384641 24954 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.402824 25028 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.419397 24852 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:02.421048 25099 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.421459 24995 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6277899576264e68aa43514e5b5c676e. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.422498 26029 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.425060 24954 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.426335 26029 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.439483 26062 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.447389 25099 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.457667 25100 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.459669 24853 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:02.485332 25028 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.486357 25069 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9910a5336aa4e3cab11dca4a556fabf. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.486891 26018 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.487866 24954 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.491672 24951 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.493253 26053 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.501196 26052 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.502843 25028 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.503041 24951 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.519569 24853 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.800745 26067 ts_tablet_manager.cc:902] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:02.804800 26067 tablet_copy_client.cc:308] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:02.806668 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 6277899576264e68aa43514e5b5c676e from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:02.807210 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 6277899576264e68aa43514e5b5c676e from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e
I20240426 13:19:02.812971 25041 tablet_copy_source_session.cc:205] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.815369 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.828485 26067 tablet_copy_client.cc:783] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.829033 26067 tablet_copy_client.cc:647] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.833396 26067 tablet_copy_client.cc:523] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.840179 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:02.857419 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:02.858219 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:02.858826 26067 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.005s
I20240426 13:19:02.861249 26067 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.861819 26067 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:02.862378 26067 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.865396 26067 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.005s
I20240426 13:19:02.867390 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:02.867842 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e on tablet 6277899576264e68aa43514e5b5c676e with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:02.869371 26074 ts_tablet_manager.cc:902] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:02.875453 26074 tablet_copy_client.cc:308] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:02.876981 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet c896b58b7766401eac0a7f69574827f1 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:02.877388 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet c896b58b7766401eac0a7f69574827f1 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1
I20240426 13:19:02.883700 25114 tablet_copy_source_session.cc:205] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.886911 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.898563 26074 tablet_copy_client.cc:783] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.899091 26074 tablet_copy_client.cc:647] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.902982 26074 tablet_copy_client.cc:523] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.906081 26067 ts_tablet_manager.cc:902] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:02.912423 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:02.936269 26067 tablet_copy_client.cc:308] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:02.938190 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 256415e42b6c444387115ab766a5a16c from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:02.938753 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 256415e42b6c444387115ab766a5a16c from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c
I20240426 13:19:02.948165 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:02.949070 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:02.949703 26074 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.014s
I20240426 13:19:02.950698 24966 tablet_copy_source_session.cc:205] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.953358 26074 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.954231 26074 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:02.954515 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.954881 26074 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.957536 26074 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20240426 13:19:02.958961 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:02.959333 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1 on tablet c896b58b7766401eac0a7f69574827f1 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:02.974996 26067 tablet_copy_client.cc:783] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.976105 26067 tablet_copy_client.cc:647] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.979594 26067 tablet_copy_client.cc:523] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.989573 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:03.008121 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:03.008913 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:03.009466 26067 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.003s
I20240426 13:19:03.011767 26067 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.012408 26067 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:03.013116 26067 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.017791 26067 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20240426 13:19:03.019201 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:03.019588 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c on tablet 256415e42b6c444387115ab766a5a16c with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:03.061695 26067 ts_tablet_manager.cc:902] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:03.065271 26067 tablet_copy_client.cc:308] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:03.066452 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet b9910a5336aa4e3cab11dca4a556fabf from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:03.066872 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet b9910a5336aa4e3cab11dca4a556fabf from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf
I20240426 13:19:03.076575 25114 tablet_copy_source_session.cc:205] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:03.079156 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:03.090981 26067 tablet_copy_client.cc:783] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:03.091554 26067 tablet_copy_client.cc:647] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:03.100021 26067 tablet_copy_client.cc:523] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:03.108280 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:03.125591 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:03.126271 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:03.126857 26067 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.001s
I20240426 13:19:03.128942 26067 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.129686 26067 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:03.130108 26067 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.131898 26067 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240426 13:19:03.134406 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:03.134795 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf on tablet b9910a5336aa4e3cab11dca4a556fabf with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:04.197486 26099 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.198417 26099 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.201028 26099 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.201491 26098 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.202109 26098 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.206020 26098 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.247025 26102 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.247726 26102 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.252483 26102 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.254750 26101 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.255283 26101 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.257158 26101 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.273586 26106 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.274185 26106 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.288235 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.288800 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1080) took 1061 ms. Trace:
I20240426 13:19:04.288947 26103 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.289484 26103 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.290330 26106 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.289299 25023 rpcz_store.cc:276] 0426 13:19:03.227760 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:03.227949 (+   189us) service_pool.cc:225] Handling call
0426 13:19:03.348179 (+120230us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:03.348371 (+   192us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:03.348399 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:03.348447 (+    48us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:03.348480 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:04.288543 (+940063us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:04.288708 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:04.289501 26101 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.294337 26096 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.295020 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1081) took 1065 ms. Trace:
I20240426 13:19:04.294939 26096 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.295293 25028 rpcz_store.cc:276] 0426 13:19:03.229935 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:03.230055 (+   120us) service_pool.cc:225] Handling call
0426 13:19:04.294545 (+1064490us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:04.294702 (+   157us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:04.294815 (+   113us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:04.294831 (+    16us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:04.294856 (+    25us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:04.294875 (+    19us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:04.294912 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:04.294980 (+    68us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4982528}
I20240426 13:19:04.291577 26103 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.297641 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:04.304328 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.0225e+06us
I20240426 13:19:04.305758 26106 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.306264 26106 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.308394 26106 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.311071 26106 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.311612 26106 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.314294 26106 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.315784 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.316442 25104 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:04.316767 26100 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.316932 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.317400 26100 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.292235 26101 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.320621 26100 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.322876 26100 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.323536 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.323360 26100 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.324473 26101 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.325408 26100 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.326151 26100 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.326680 26100 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.328702 26100 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.329471 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.296869 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.301607 26098 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.302075 26097 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.331400 26097 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.331462 26098 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.333216 26101 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.333348 26097 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.302997 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.333711 26101 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.299479 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.318684 25621 raft_consensus.cc:1203] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:04.334723 24953 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.318881 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:04.338172 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.297847 26096 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.335785 26101 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.345084 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.345180 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.344383 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.345489 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.342612 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.348901 25620 raft_consensus.cc:1203] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:04.347630 26107 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.347735 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.356138 26096 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.345916 26106 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.362097 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.362134 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.362860 26096 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.364224 26107 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.374132 26096 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.374284 26107 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.389904 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.390844 25099 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.360368 26106 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.470813 26095 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.471333 26095 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.486956 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.465292 26106 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.461514 26107 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.497352 26103 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.496608 25618 raft_consensus.cc:1203] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:04.464810 26102 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.470120 26098 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.469780 26097 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.522184 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:04.485286 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 140220us
I20240426 13:19:04.548730 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.554600 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 201690us
I20240426 13:19:04.519831 26112 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.563189 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:04.525398 26103 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.518607 26095 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.521283 26113 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:04.569607 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 205988us
I20240426 13:19:04.523006 26107 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.575383 26114 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.525080 26102 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.571854 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.576015 26104 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.576542 26104 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.578939 26103 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.569164 26098 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:04.567718 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 205688us
I20240426 13:19:04.582087 26104 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.582321 26102 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.562381 26097 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.597486 26115 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.598101 26109 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.570482 26112 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.573710 26101 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.581295 26098 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.600384 26116 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.608480 26119 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.608826 26109 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.612102 26109 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.601253 26104 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.613278 26104 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.615643 26107 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.616248 26107 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.617113 26112 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.616818 26107 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.617903 26117 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.605686 26098 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.643393 26112 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.644124 26112 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.646576 26112 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.672863 26112 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.673579 26112 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.618387 26117 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.607510 26115 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.609061 26119 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.619180 26114 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.704139 26118 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.574576 26113 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.605481 26101 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.747735 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.757802 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.711843 26117 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.710330 26115 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.704849 26106 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.762382 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:04.766434 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.740432 26116 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.769995 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.743670 26110 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:04.774116 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75082.6us
I20240426 13:19:04.775053 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.776927 26112 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.777052 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.777601 26101 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.772768 26117 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.798383 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.712738 26114 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.742097 26118 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.777601 26110 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.778503 26104 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.772989 26116 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.782083 26094 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.877566 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.936910 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.904229 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 141961us
I20240426 13:19:04.942623 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.943544 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.834307 26106 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.781936 25615 raft_consensus.cc:1203] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:04.944407 25618 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.946386 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.805853 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.947501 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.947144 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.902905 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 304936us
I20240426 13:19:04.874433 26126 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.949040 26126 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.950109 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.951393 26127 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.951610 26126 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.875815 26119 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.879195 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.952863 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.953190 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.877506 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:04.960424 26132 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.961084 26132 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.833194 26123 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.961558 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.953505 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.969955 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.970152 25618 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.874239 26128 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.880482 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.971071 26128 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.878769 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.973205 26128 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.955242 25620 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:04.955320 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.956434 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.957830 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.975557 25097 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.975513 25618 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:04.798013 26097 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.959303 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:04.834525 26121 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.981076 26130 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.980950 26121 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.981336 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:04.981536 26130 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.981971 26130 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.983045 26121 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.877686 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:04.984534 26092 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.962885 26123 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.963299 26132 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.833459 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.985412 25619 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:04.964253 26104 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.986060 26104 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.986685 26123 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.987915 26104 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.803085 26117 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:04.989609 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 221209us
I20240426 13:19:04.990131 26117 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.966044 26110 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:04.896981 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 132061us
I20240426 13:19:04.891834 26113 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.806124 26122 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.991391 26122 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.991904 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.992563 24956 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:04.993484 26122 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.993767 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.994951 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:04.997022 26102 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.997409 26102 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.997758 26102 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.003026 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:05.003587 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.004240 26117 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.004516 26111 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.004619 26117 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.005117 26117 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.005793 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.006520 26111 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.009742 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.881662 26115 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.010572 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:05.011845 26111 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.012290 26111 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.012715 26111 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.013355 26130 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.015426 26130 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.019389 26111 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.021054 26100 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:04.968753 26114 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.968765 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.805580 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.971100 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.009610 26102 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.026748 26102 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.008379 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.023036 24948 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:05.011071 26115 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.940670 26107 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.024711 26134 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.025022 26116 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.031867 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:04.975502 26118 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.036358 26117 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.036932 26114 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.053326 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45100.9us
I20240426 13:19:05.021598 26111 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.054680 26134 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:05.054782 26094 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.058730 26094 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:05.058782 24950 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:05.059186 26094 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.054913 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:05.056047 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85013.7us
I20240426 13:19:05.056874 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.068328 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.063841 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.068565 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.063967 26123 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.077443 26123 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.077828 26123 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.079181 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:05.079353 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.086117 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:05.086736 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.086879 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.085034 26133 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.088186 26133 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.084996 26110 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.089735 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:05.090602 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63218.8us
I20240426 13:19:05.094173 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.095098 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.095329 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.097043 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.097250 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.088508 26107 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.099107 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.102414 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.103227 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.088538 26133 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.107007 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.091318 26092 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.112282 26092 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.099963 26117 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.102340 26107 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.119838 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.120293 26115 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.120473 26092 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.120391 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.121261 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.121834 26107 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.089098 26098 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.110356 25615 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.122169 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.153338 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.151949 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.153512 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.152341 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.117486 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.156304 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.158493 25031 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.157912 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.161355 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.161813 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.162856 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.157490 26117 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.161841 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.165253 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.164288 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.166098 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.166764 26098 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.151813 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.171029 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.171154 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.171427 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.176658 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.181100 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.181895 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.183096 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.184651 25069 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.185482 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:05.169941 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52575.1us
I20240426 13:19:05.186136 26100 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.186597 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.187124 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.187186 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.187821 26104 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.191507 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.188081 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.187842 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:05.188689 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080013d280 after lost signal to thread 23799
I20240426 13:19:05.193360 26135 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.189188 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.193562 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.194538 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.195216 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.191049 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.190086 26098 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.196174 26104 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.196586 26098 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:05.196969 26098 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:05.196257 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08001754c0 after lost signal to thread 24888
I20240426 13:19:05.198083 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.199718 26125 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.199764 26123 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.201100 25624 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.201773 26123 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:05.202243 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08002b0d40 after lost signal to thread 24980
I20240426 13:19:05.203034 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:05.203295 25126 debug-util.cc:398] Leaking SignalData structure 0x7b0800272720 after lost signal to thread 25055
I20240426 13:19:05.204810 26117 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.204852 26104 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.205013 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.113448 26134 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.206670 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.207787 26134 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.208266 26101 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.209496 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.172238 26133 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.210749 26119 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.213763 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.215905 26129 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.217235 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.219221 26101 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.219333 26093 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.219887 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf0c0 after lost signal to thread 25129
I20240426 13:19:05.220256 26135 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.220991 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080009e2a0 after lost signal to thread 25650
I20240426 13:19:05.222211 26107 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.223239 24952 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.226193 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.230022 26117 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.230360 26117 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.230691 26117 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.230561 26104 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.237509 26100 raft_consensus.cc:1050] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.239755 26100 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.240198 26117 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.240850 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.242239 26135 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.226099 26094 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.243340 26104 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.243738 26104 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.242417 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.244740 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.244518 26104 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.246683 26117 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.247110 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.247480 26104 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.248210 26133 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.248553 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.249451 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.249054 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.250605 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.259408 25028 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.257059 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.253021 26124 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.257530 25025 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.268652 26124 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.252856 25624 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.261755 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.270200 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.272209 26094 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.272445 25619 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.273538 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.274883 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.269735 26124 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.276172 26109 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.276983 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.278170 26109 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.278234 26110 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.279237 26135 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.280701 26098 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:05.282033 25026 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.284811 26098 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.270570 26107 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.286746 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.288700 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.290055 26135 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.291103 26115 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:05.292481 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.292603 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.291226 26125 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:19:05.293805 26135 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.294421 26119 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.252667 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.294903 26119 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.295228 26117 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.295565 26125 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.295974 26125 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.294379 26108 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.295711 26119 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.297688 26119 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.299124 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.297313 26125 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.299649 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.300500 26125 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.301487 26129 raft_consensus.cc:1050] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.301261 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.299147 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.302608 26117 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.302894 26101 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 13:19:05.291591 26124 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.304750 26135 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:05.305936 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.309085 26135 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.309427 26135 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.310194 26135 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.313758 26135 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.315625 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.313959 26129 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.323382 25615 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.325311 26094 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:05.328759 24950 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.331544 26094 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.333865 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.336683 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.337495 25096 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:05.338994 25103 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.342873 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.343856 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.344794 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.345798 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.346869 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.347250 26125 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.347820 26125 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.348608 26094 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.348893 26098 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.349802 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.346897 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.349403 26125 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.351011 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.352607 26117 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.352885 26101 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.353055 26117 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:05.353996 26117 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.355979 26133 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.358808 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.360149 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.360903 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.364235 25105 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.364362 26125 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.366843 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.368573 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.376741 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.381930 26120 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.383155 26097 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.386070 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.383384 26107 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.387794 26107 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.388494 25101 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:05.388657 26107 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.395705 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.396915 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.397231 25096 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.385555 26119 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.400210 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.401202 26099 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.402666 26123 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.403136 26123 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:05.403939 26123 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.396577 26100 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.408861 26119 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.410733 26099 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.423230 25097 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:05.440147 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:05.446139 26107 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.447141 26097 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.448570 24950 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.446821 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.454563 26100 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.456223 26100 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:05.458148 26100 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.457779 25101 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.473368 24851 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.476925 25615 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.490265 24852 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:05.509581 26112 raft_consensus.cc:1050] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.511970 26112 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.547250 25615 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.552986 26112 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.563880 25023 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.565464 26124 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.565752 25619 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.566442 26124 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.566769 26124 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.569741 24950 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.580518 26121 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:05.581614 26121 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.581929 26121 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.582690 26121 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.586216 26121 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.593179 26132 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.596801 25023 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.598212 26104 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.605584 26104 raft_consensus.cc:1050] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.605988 26124 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.608055 26104 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.613466 26124 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.613687 24854 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:05.633024 25023 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.634346 26132 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.637847 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.638185 24956 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.644657 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.645064 25026 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.645443 26112 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.646404 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.651007 25619 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.655570 24950 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.657022 26136 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.662333 25615 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.664379 26098 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.668812 25030 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.670087 24949 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.675109 26136 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.677785 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6277899576264e68aa43514e5b5c676e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.677933 24854 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.682184 26104 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.687268 25619 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.694911 25023 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.698279 24952 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.704141 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:05.702427 25620 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.707754 24949 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.716522 25026 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.718809 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 6277899576264e68aa43514e5b5c676e with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:05.724812 26124 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.726094 26124 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.726522 26124 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.727998 26124 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.739967 26159 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:05.748867 26159 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:05.749612 26159 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:05.750974 25101 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.756578 24851 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.760147 24956 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.760371 25619 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.761933 26112 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.762913 26104 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.769004 26159 ts_tablet_manager.cc:1875] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:05.770622 26104 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.777133 25619 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.794201 24956 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.795786 26159 ts_tablet_manager.cc:1888] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:05.804332 26159 log.cc:1196] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/6277899576264e68aa43514e5b5c676e
I20240426 13:19:05.798082 26124 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.805929 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 6277899576264e68aa43514e5b5c676e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:05.817647 25101 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.840247 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9910a5336aa4e3cab11dca4a556fabf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.842514 25026 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.847887 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.850582 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.852121 26112 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.853863 26112 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:05.853358 24851 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.854373 26112 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.857168 26103 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.858479 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.859082 25030 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:05.861056 25030 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.864929 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet b9910a5336aa4e3cab11dca4a556fabf with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:05.862490 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:05.867993 26109 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.868366 26109 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:05.868738 26109 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.881449 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.882298 25620 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.882382 26109 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.889909 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:05.891209 25620 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.893210 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.894002 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.896456 26109 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.903261 26160 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:05.905205 26160 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:05.905503 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:05.906008 26160 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:05.907784 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:05.908610 24956 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:05.909659 26121 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.909963 26123 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.910429 26121 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.911356 26121 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.914669 26160 ts_tablet_manager.cc:1875] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:05.917899 24950 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:05.919531 26123 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.919837 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c896b58b7766401eac0a7f69574827f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.922292 24854 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.928411 26112 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.004139 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet c896b58b7766401eac0a7f69574827f1 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:06.019820 25105 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.020599 25097 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.023087 26123 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.024305 26123 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:06.027088 26160 ts_tablet_manager.cc:1888] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:06.027675 26160 log.cc:1196] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/b9910a5336aa4e3cab11dca4a556fabf
I20240426 13:19:06.029448 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet b9910a5336aa4e3cab11dca4a556fabf (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:06.030599 25101 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:06.032519 26098 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:06.040599 24851 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:19:06.047163 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45923.5us
I20240426 13:19:06.053130 24947 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:06.081980 24947 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:06.092685 26166 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
W20240426 13:19:06.099574 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45881.7us
I20240426 13:19:06.102301 26166 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
W20240426 13:19:06.114255 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52488.7us
I20240426 13:19:06.127043 26099 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240426 13:19:06.144543 26166 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:19:06.156086 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59401.5us
I20240426 13:19:06.187179 26166 ts_tablet_manager.cc:1875] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:06.257426 26166 ts_tablet_manager.cc:1888] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:06.257818 26166 log.cc:1196] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/c896b58b7766401eac0a7f69574827f1
I20240426 13:19:06.259346 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet c896b58b7766401eac0a7f69574827f1 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:06.356907 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.358153 26109 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.372047 25030 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.377082 26121 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.506356 26181 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.506820 26181 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.508610 26181 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:06.509614 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.510170 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:06.510185 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:06.512280 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:06.513016 26181 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.513342 26181 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.513583 26181 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.520750 26181 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.522248 26181 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:06.522855 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.523581 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:06.523784 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.530593 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.532351 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:06.533391 26181 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.534034 26181 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:06.534823 26181 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.547057 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:06.556799 26121 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.558270 26121 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.563107 26121 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.563948 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.565342 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:06.566668 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:06.567291 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.568243 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:06.570348 26103 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:06.570708 26103 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.571070 26103 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:06.580511 26103 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.584290 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.584919 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:06.585716 26103 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.589048 26112 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.589601 26112 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.590543 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:06.608073 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:06.612100 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.612912 25026 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.613963 25070 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.614256 25070 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:06.614723 26103 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.617380 24956 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.620816 26103 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.627768 26119 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.628209 26119 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.631383 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:06.632651 26109 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:06.633417 26109 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:06.634229 26109 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.638338 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.639042 25026 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:06.640545 26119 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.641564 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:06.649267 26112 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.666720 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:06.667385 25103 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:06.674460 24853 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:06.684571 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:06.686204 26181 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.686616 26181 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.686936 26181 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.697841 26181 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.703752 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.704341 25030 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.710153 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.713415 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:06.714040 25101 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.722888 25101 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.724761 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:06.727402 26181 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.727548 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:06.725486 26119 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.728722 26119 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:06.729497 26119 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.745703 24853 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:06.775959 26189 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.777247 26189 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.779757 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.780406 25030 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.789108 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:06.788084 26189 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.788117 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:06.805605 26189 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.805994 26189 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.806339 26189 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.813710 26189 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.815898 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.816502 25026 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.821709 25026 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:06.823226 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:06.824092 26189 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.823957 26103 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.825001 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:06.825196 26103 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:06.826145 26103 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.846872 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:07.000401 25030 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:07.001623 26119 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.023224 26103 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:07.023728 26103 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.025718 26103 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.027524 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.028142 25620 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:07.027544 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:07.030539 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:07.031512 26109 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:07.031854 26109 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:07.032178 26109 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.033342 25103 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:07.035144 25030 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:07.037431 26119 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.037771 26103 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.040403 26109 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.042647 26172 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.043145 26172 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.049827 26172 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.051761 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.053772 25101 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:07.055426 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:07.056440 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:07.057228 25025 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:07.057356 26181 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.057813 26181 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.060221 26123 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:07.060532 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.060722 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.061036 26172 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.061190 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:07.061336 25620 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:07.061542 26172 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.063318 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.063879 25101 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.070784 24921 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.071080 24921 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:07.071640 26119 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.073745 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:07.074623 26172 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.063488 26181 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.077723 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.079810 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:07.084487 25025 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.089252 26109 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.091631 25589 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.091989 25589 leader_election.cc:401] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:07.093098 26172 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.094566 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:07.095155 25620 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.095306 26119 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.102582 24949 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:07.110772 25620 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:07.111886 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:07.114450 26109 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:07.118217 26109 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:07.122454 26109 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.085487 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.130755 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:07.131026 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.131536 25023 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:07.137828 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:07.153720 24950 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:07.155480 26189 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.160181 25025 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20240426 13:19:07.381871 26195 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.233s	user 0.000s	sys 0.000s
I20240426 13:19:07.386315 26172 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.402413 25096 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:07.450693 25619 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:07.454936 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 283740us
I20240426 13:19:07.461310 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:07.462574 26172 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:07.463050 26172 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:07.463872 26172 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.478096 26179 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:07.478781 26179 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.479569 26202 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:07.519127 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:07.605753 26179 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.606576 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.622138 25615 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Advancing to term 3
I20240426 13:19:07.645623 25101 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:07.646405 25615 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:07.674978 26212 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:07.688021 26205 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:07.719703 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:07.733361 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53838.4us
I20240426 13:19:07.720736 25101 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:07.989724 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:07.996188 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62466.3us
I20240426 13:19:07.998689 24947 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:08.004899 26224 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:08.005302 26224 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:08.005604 26224 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:08.005702 24854 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:08.013028 26224 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.014969 26224 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:08.035773 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.045s	sys 0.126s
I20240426 13:19:08.036115 25624 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:08.036375 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.046s	sys 0.126s
I20240426 13:19:08.038326 26212 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:08.071712 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:08.075572 25103 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240426 13:19:08.078553 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61597.9us
I20240426 13:19:08.097368 26172 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:08.131182 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.132192 25101 raft_consensus.cc:595] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.132717 26229 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:08.135600 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.136206 25097 raft_consensus.cc:595] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.149912 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:19:08.165073 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 150369us
I20240426 13:19:08.158391 25624 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:08.179548 25097 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:08.180685 25022 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:08.188545 26121 raft_consensus.cc:979] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.191682 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.148339 26109 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240426 13:19:08.189050 26121 raft_consensus.cc:1067] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.258165 25023 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:08.131120 26211 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:19:08.270009 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66457.3us
I20240426 13:19:08.280381 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:08.280597 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 110361us
W20240426 13:19:08.276211 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73893.8us
W20240426 13:19:08.302747 26234 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20240426 13:19:08.291883 26213 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:08.340989 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64393.9us
I20240426 13:19:08.343602 26247 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:08.381039 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:08.328512 26246 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20240426 13:19:08.416464 26229 raft_consensus.cc:979] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.416934 26229 raft_consensus.cc:1067] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.396167 26247 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:08.423687 25620 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:08.424324 25620 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:08.424700 25620 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.441051 25620 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.414335 25615 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:08.456233 25620 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.446614 25615 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:08.475461 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46494.5us
W20240426 13:19:08.487648 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 216965us
I20240426 13:19:08.504118 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:08.504760 25031 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.498029 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.514169 25097 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:08.514513 25097 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:08.515223 25097 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.515859 25615 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.516456 25097 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.517681 25615 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.515653 26247 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.512689 26123 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:08.567473 26123 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:08.568321 26123 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.569626 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:08.574152 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62035.9us
I20240426 13:19:08.574173 24955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:08.581348 24955 raft_consensus.cc:595] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.587365 25097 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:08.608737 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:08.614356 25031 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:08.641170 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:08.660149 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:08.688361 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.688889 25101 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:08.689666 25101 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:08.676769 25022 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:08.696764 25031 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20240426 13:19:08.698724 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50442.1us
I20240426 13:19:08.690387 25101 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.706107 25101 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:08.708711 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55531.6us
I20240426 13:19:08.712556 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.713145 25102 raft_consensus.cc:595] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.714807 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:08.756503 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48369.7us
W20240426 13:19:08.758169 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016fe60 after lost signal to thread 23799
W20240426 13:19:08.759112 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d540 after lost signal to thread 24888
W20240426 13:19:08.760491 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080025dde0 after lost signal to thread 24980
W20240426 13:19:08.761816 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08003660c0 after lost signal to thread 25055
W20240426 13:19:08.763152 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08002a4ce0 after lost signal to thread 25129
I20240426 13:19:08.766374 26172 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:08.766898 26172 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:08.767791 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080032f660 after lost signal to thread 25650
I20240426 13:19:08.796883 25101 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:08.772344 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:08.809863 25024 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:08.767693 26172 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.875814 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:19:08.889847 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174729us
I20240426 13:19:08.849449 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:08.894796 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:08.901880 26261 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:08.905643 26172 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:08.907351 26172 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:08.908203 26172 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:08.900869 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63071.8us
W20240426 13:19:08.985321 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78980us
I20240426 13:19:08.990947 26243 raft_consensus.cc:979] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.007002 26109 raft_consensus.cc:979] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.010993 26109 raft_consensus.cc:1067] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.013327 26205 raft_consensus.cc:1067] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.019471 25022 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.027599 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:09.028340 25615 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.028700 25615 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.039350 25615 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.042286 25615 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.043615 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.044301 25098 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.044799 25098 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:09.045622 25098 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.046958 25098 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.052728 25098 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.053591 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:09.054204 26172 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.054562 26172 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.055140 26172 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.103628 25024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:09.106316 25024 raft_consensus.cc:595] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:09.126919 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:09.131350 25615 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.137097 25615 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.136757 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:09.139248 24956 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.138242 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:09.140415 25096 raft_consensus.cc:595] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:09.142915 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:19:09.144610 24852 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.153580 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:09.155416 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:09.218394 26240 raft_consensus.cc:979] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.219018 26240 raft_consensus.cc:1067] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.232475 24956 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.248474 25096 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:09.250388 26172 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240426 13:19:09.266252 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:09.266942 24951 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.267335 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.283876 26228 raft_consensus.cc:979] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.283540 24948 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.284345 26228 raft_consensus.cc:1067] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.286115 24954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:09.286624 24954 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.287006 24954 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.294111 25030 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:09.296895 24951 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.301457 26123 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 13:19:09.304256 25098 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.304893 25022 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.313103 26250 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:09.314509 26250 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.311196 24954 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.329138 25615 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.330534 25623 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:09.349640 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:09.350071 25024 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:09.352723 24951 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.339080 25615 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:09.350378 24956 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.354635 24956 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.361096 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.362010 25025 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.362237 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.362473 25025 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:09.362931 25026 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.366456 26242 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20240426 13:19:09.372720 25619 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:09.374603 26242 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.379547 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.380196 25104 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.386173 24956 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.395342 24956 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.363655 25025 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.406592 25025 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.414319 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.417788 24954 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.416967 26212 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.428727 24956 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.429203 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:09.395431 26205 consensus_queue.cc:1159] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1590f80bdade41c6a755b8501c55fe16" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:09.432981 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:09.434605 26243 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.434670 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.435287 26243 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.435518 25029 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.436282 26243 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.434311 25026 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.453224 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.453804 25104 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.454048 25104 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:09.454691 25104 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.405850 24854 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.444368 25031 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.458604 24854 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.464879 24956 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.465927 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:09.469033 26250 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.468420 25619 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.469678 26250 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.470908 26250 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.474584 26172 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:09.506109 25104 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.515175 25029 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.523396 25104 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.534922 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:09.547672 26181 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.548169 26181 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.553416 26181 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.592749 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 256415e42b6c444387115ab766a5a16c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240426 13:19:09.604398 24839 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55525.2us
I20240426 13:19:09.668256 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.705345 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.759604 25104 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240426 13:19:09.792160 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 153784us
I20240426 13:19:09.799228 24854 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:09.809404 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 256415e42b6c444387115ab766a5a16c with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:09.815708 26172 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.825088 24854 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:09.835853 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:09.837730 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.839756 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.841379 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:09.842113 25022 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:09.843253 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:09.843992 26123 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:09.844475 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:09.846072 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.851378 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:09.869210 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.872817 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.873406 25022 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.879391 24948 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.880184 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.884546 26250 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.886287 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.887671 25096 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:09.871304 26295 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:09.889017 26292 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:09.885164 25022 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:09.912144 25025 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.913605 26123 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.918265 25030 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:09.920224 26255 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:09.928977 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:09.931910 26287 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:09.932406 26287 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.933293 26287 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.949069 26295 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:09.950117 26295 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:09.952435 26229 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:09.953045 26229 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.955267 26229 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:09.956583 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:09.958261 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:09.958917 25030 raft_consensus.cc:2354] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.960575 25070 leader_election.cc:398] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.960932 25070 leader_election.cc:401] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:09.961730 26229 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.969462 26295 ts_tablet_manager.cc:1875] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:09.976542 26229 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.989490 26295 ts_tablet_manager.cc:1888] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240426 13:19:09.990077 26295 log.cc:1196] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/256415e42b6c444387115ab766a5a16c
I20240426 13:19:09.991875 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 256415e42b6c444387115ab766a5a16c (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:10.001513 25022 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:10.003198 26181 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:10.009047 25104 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:10.012120 26181 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:10.013339 24853 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:10.440588 25025 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:10.442196 26250 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:10.455636 25104 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:10.457608 26287 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:10.908497 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:10.969285 24956 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:19:10.987167 24920 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c997ccc9a12742c3b149dc9747b2a0ae. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:10.990464 25104 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:10.990464 25025 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:10.992363 26244 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.002318 26255 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.004875 26292 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.019039 25022 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.021293 25030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:11.021896 25030 raft_consensus.cc:595] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:11.010464 25104 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.030915 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9911e0422347463c8781b2574c023744"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:11.031478 25098 raft_consensus.cc:595] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:11.034477 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:11.035112 25104 raft_consensus.cc:595] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:11.043256 26229 raft_consensus.cc:979] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:11.043713 26229 raft_consensus.cc:1067] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:11.045837 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:11.045200 24853 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.047591 24956 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:11.046381 25025 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.048692 25025 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.056070 25025 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.070996 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.071238 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:11.071611 25104 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.071869 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.071961 25104 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.072791 25104 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.073899 25104 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.080170 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:11.081367 25025 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:11.082470 25104 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:11.083951 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:11.084280 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:11.084970 24948 raft_consensus.cc:595] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:11.084947 26311 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.086583 26311 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:11.087680 24920 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53343363b25040c29f4c212b440dc2cc. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:11.088758 25104 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:11.090222 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:11.090284 26307 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.092288 26244 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.095448 25025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:11.096150 25025 raft_consensus.cc:595] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:11.098066 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:11.099864 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:11.100656 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:11.087553 26311 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.126197 26244 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.133296 25098 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.134692 25022 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.150013 25104 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:11.175115 25022 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:11.177516 26248 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:19:11.181125 25069 consensus_peers.cc:485] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 764a1a5c07f94909b2fd69a50de4bedd. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:11.182650 24948 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:11.188505 26267 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:11.195638 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.210300 26244 raft_consensus.cc:979] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:11.210947 26255 raft_consensus.cc:1067] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:11.213351 25623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:11.213933 25623 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.214237 25623 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.239013 24853 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.248667 26311 raft_consensus.cc:979] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.249174 26311 raft_consensus.cc:1067] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.244819 25623 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.250881 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:11.253582 24956 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.254037 24956 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.255352 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:11.255930 24954 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.256225 24954 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.256896 24954 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.258416 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.259020 25025 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.257930 24954 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.260116 25623 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:11.269012 25025 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:11.280974 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:11.281344 24956 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.282049 26287 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.280941 24954 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:11.283581 26287 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:11.284785 26287 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.286314 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.285305 24956 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.286903 25104 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.286321 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.287577 25030 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.287881 25030 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:11.289031 25030 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.290261 25030 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.295622 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.296067 26229 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.298817 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:11.305336 26244 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.305788 26244 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.306622 26244 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.311275 25022 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.315057 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:11.317328 24948 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.329857 25030 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:11.341734 24901 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20240426 13:19:11.342902 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:19:11.363369 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9911e0422347463c8781b2574c023744, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:11.364106 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9911e0422347463c8781b2574c023744, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.408951 26267 raft_consensus.cc:979] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.409431 26267 raft_consensus.cc:1067] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.413873 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9911e0422347463c8781b2574c023744"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:11.431824 24853 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.416887 24948 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.455490 24948 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.466836 26323 ts_tablet_manager.cc:902] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:11.475250 24948 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.478063 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.478736 25022 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.497128 25022 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.498667 24948 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.500926 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:11.502640 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.511093 25104 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.511437 25104 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.512391 25104 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.513819 25104 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.515607 26253 raft_consensus.cc:979] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:11.516124 26253 raft_consensus.cc:1067] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:11.518143 25096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58860
I20240426 13:19:11.518743 25096 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.519090 25096 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.524868 26255 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.525444 26255 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.526300 26255 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.566710 25098 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.571921 26291 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:11.580655 25096 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.612738 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.636229 26253 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:11.658488 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:11.659168 24954 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.669584 26332 ts_tablet_manager.cc:902] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:11.672082 24954 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:11.673346 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:11.674247 26229 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.674724 26229 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.675592 26229 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.695578 25104 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.698784 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.706530 25030 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.706977 25030 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:11.721374 25030 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.728581 25030 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.729436 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.713075 25096 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:11.733331 24851 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:11.743494 25030 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:11.768783 24948 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.779429 26287 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240426 13:19:11.861213 25022 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.861325 25030 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:11.869308 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76510us
I20240426 13:19:11.870509 26342 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:11.871730 26341 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:11.881803 26343 ts_tablet_manager.cc:902] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.892318 25097 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:11.912863 26344 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240426 13:19:11.888123 26332 tablet_copy_client.cc:308] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:11.929453 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 53343363b25040c29f4c212b440dc2cc from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:11.945912 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 53343363b25040c29f4c212b440dc2cc from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc
I20240426 13:19:11.962152 24966 tablet_copy_source_session.cc:205] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:11.925999 26323 tablet_copy_client.cc:308] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:11.984999 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:19:12.000450 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87783.1us
W20240426 13:19:12.002002 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80479us
I20240426 13:19:12.005722 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet c997ccc9a12742c3b149dc9747b2a0ae from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:12.006309 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet c997ccc9a12742c3b149dc9747b2a0ae from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae
I20240426 13:19:12.014372 24966 tablet_copy_source_session.cc:205] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:12.072718 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:19:12.090535 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45430.9us
I20240426 13:19:12.093142 25030 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.098063 26255 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.145236 25097 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.166028 26332 tablet_copy_client.cc:783] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.166673 26332 tablet_copy_client.cc:647] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.179445 26323 tablet_copy_client.cc:783] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.182430 26307 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:19:12.180076 26323 tablet_copy_client.cc:647] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.190698 26323 tablet_copy_client.cc:523] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.214933 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.220314 26229 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.239074 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.251957 24956 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.254038 26354 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.272836 26332 tablet_copy_client.cc:523] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.280162 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:19:12.300278 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63772.7us
I20240426 13:19:12.339043 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.358680 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.359499 26323 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.121s	user 0.022s	sys 0.003s
I20240426 13:19:12.356315 26343 tablet_copy_client.cc:308] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:12.365880 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.362656 26323 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.380142 26323 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.380390 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 764a1a5c07f94909b2fd69a50de4bedd from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:12.381031 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 764a1a5c07f94909b2fd69a50de4bedd from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd
I20240426 13:19:12.381901 26323 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.392995 24954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:12.393612 24954 raft_consensus.cc:595] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:12.401533 25114 tablet_copy_source_session.cc:205] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:12.402160 24954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:12.411176 24954 raft_consensus.cc:595] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:12.412750 26343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:12.402201 26323 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.004s
I20240426 13:19:12.425189 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:12.425621 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae on tablet c997ccc9a12742c3b149dc9747b2a0ae with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.427431 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:12.431526 25097 raft_consensus.cc:595] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:12.437207 26343 tablet_copy_client.cc:783] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.437781 26343 tablet_copy_client.cc:647] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.437860 25025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:12.438705 25025 raft_consensus.cc:595] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:12.440552 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:12.442672 26343 tablet_copy_client.cc:523] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.442880 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:12.451213 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:12.473702 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.474834 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.478408 26332 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.113s	user 0.021s	sys 0.010s
I20240426 13:19:12.486958 26332 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.489441 26332 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.490154 26332 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.497483 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.513144 26332 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20240426 13:19:12.535058 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:12.535492 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc on tablet 53343363b25040c29f4c212b440dc2cc with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.550453 26255 raft_consensus.cc:979] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.550951 26255 raft_consensus.cc:1067] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.553077 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:12.553679 25025 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.554015 25025 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.563433 25623 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20240426 13:19:12.601352 25025 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.605789 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:12.606438 24948 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:12.606736 24948 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.607406 24948 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.608409 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.615164 26244 raft_consensus.cc:979] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.615664 26244 raft_consensus.cc:1067] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.617190 25029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:12.617760 25029 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.618095 25029 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.625341 25029 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.631649 25029 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:12.634541 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:12.635087 24949 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:12.635392 24949 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.636080 24949 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.635404 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.637825 25098 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.640780 24949 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.648262 26349 raft_consensus.cc:979] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:12.648806 26291 raft_consensus.cc:1067] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:12.651715 25098 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:12.653033 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:12.654186 26253 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:12.654824 26253 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.659613 24949 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:12.659525 26253 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.663075 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.663991 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.664672 26343 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.167s	user 0.026s	sys 0.000s
I20240426 13:19:12.668424 26343 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.668969 25097 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58860
I20240426 13:19:12.669798 25097 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.670208 25097 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.669268 26343 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.672559 26343 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.699427 25025 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:12.702380 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.707182 25096 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.718001 25096 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:12.724563 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:12.726387 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:12.731848 26291 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:12.732422 26291 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.733414 26291 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.699704 26343 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20240426 13:19:12.768860 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:12.769460 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd on tablet 764a1a5c07f94909b2fd69a50de4bedd with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.784540 26345 raft_consensus.cc:979] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:12.790813 26354 raft_consensus.cc:1067] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:12.792191 25097 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.795163 25097 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:12.796018 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:12.796731 25623 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.796018 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:12.797297 25029 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:12.797819 25029 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.798554 25029 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.799594 25029 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.801151 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.803673 25623 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:12.805167 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:12.805610 25029 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:12.805702 24956 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.806896 24956 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.806802 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:12.812052 26267 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:12.812527 26267 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:12.813369 26267 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.820565 24956 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.822396 24956 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:12.829982 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.831156 25097 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:12.831481 25097 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:12.831893 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:12.833052 25097 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.837158 25097 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.833911 25623 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.850539 25097 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:12.851706 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:12.854488 26255 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:12.854972 26255 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.855958 25623 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:12.855748 26255 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.881412 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.913084 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.913786 24851 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.973676 25623 raft_consensus.cc:1203] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240426 13:19:13.009521 26307 raft_consensus.cc:1050] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.012151 26307 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.046155 25097 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.046397 25025 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.048746 25098 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.046428 25623 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.053332 26255 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.055264 26291 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.057606 26360 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240426 13:19:13.062505 26326 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.163722 25623 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Advancing to term 4
I20240426 13:19:13.165112 26360 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.173703 25029 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.179176 25623 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.188864 24954 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.189870 25098 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.191563 26339 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.199744 26291 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.170478 25619 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.218941 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.266801 26253 raft_consensus.cc:1050] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.269140 26253 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.280615 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.281893 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.291553 25096 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.293385 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.296121 25615 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.301591 26339 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.304802 25029 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.306143 26267 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.308367 26255 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.317173 25097 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.318701 25097 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.318145 25615 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.326228 26255 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 13:19:13.331450 25619 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.333926 24956 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:13.334775 26375 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.337829 26349 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.347333 25097 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:13.348775 26349 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.358999 24949 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.370070 25096 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.385110 26307 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 13:19:13.403365 26390 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20240426 13:19:13.404811 24948 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.408461 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.411211 25615 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.411934 26253 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.420540 26326 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.436193 25098 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.439157 24949 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.476506 26255 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 13:19:13.479537 26391 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20240426 13:19:13.500178 25615 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.510890 24854 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.529825 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:13.536387 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c997ccc9a12742c3b149dc9747b2a0ae with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:13.556658 26400 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:13.557930 26400 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:13.558691 26400 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:13.561625 26400 ts_tablet_manager.cc:1875] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:13.564533 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:13.583900 26267 raft_consensus.cc:1050] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.586126 26267 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.590417 26400 ts_tablet_manager.cc:1888] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240426 13:19:13.591205 26400 log.cc:1196] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/c997ccc9a12742c3b149dc9747b2a0ae
I20240426 13:19:13.593334 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet c997ccc9a12742c3b149dc9747b2a0ae (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:13.604521 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.606376 25031 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.607646 26267 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.607913 24956 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.608937 26354 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.612468 26375 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.637253 26354 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.639189 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:13.639791 25097 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.667117 25029 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.668390 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:13.668991 24956 raft_consensus.cc:595] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:13.672211 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:13.672780 25098 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.671828 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:13.677510 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:13.678052 24956 raft_consensus.cc:595] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:13.681414 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 44835.5us
I20240426 13:19:13.684617 25619 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.692327 24956 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.696602 25029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:13.697194 25029 raft_consensus.cc:595] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.698292 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:13.699802 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:13.700382 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:13.726263 26354 raft_consensus.cc:979] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.726714 26267 raft_consensus.cc:1067] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.729445 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:13.729966 24949 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.730301 24949 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.730648 26360 raft_consensus.cc:979] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:13.731137 26398 raft_consensus.cc:1067] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:13.735158 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
 from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:13.735667 25615 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.736048 25615 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.743815 24949 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.745162 25615 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.747766 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:13.748399 24948 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.748682 24948 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.749984 25615 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:13.750990 24948 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.753187 24948 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.759547 24948 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:13.767320 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:13.768637 26287 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.769214 26287 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:13.770133 26287 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.772773 25098 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.782547 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.783248 25029 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.792171 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:13.793090 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.793390 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:13.793603 25031 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:13.793725 25030 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.794950 26375 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.796113 26354 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.793936 25098 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.797161 25098 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:13.799289 25098 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.804297 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.805382 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:13.805322 26354 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.806589 26360 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.807049 26360 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.807901 26360 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.813285 26253 raft_consensus.cc:979] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.813722 26253 raft_consensus.cc:1067] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.814086 25619 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.815603 25029 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:13.815809 25098 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.816926 25025 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.818832 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:13.823143 24948 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.823467 24948 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.834532 25098 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.836241 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 764a1a5c07f94909b2fd69a50de4bedd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:13.792210 24949 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:13.846768 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:13.855508 24948 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.872565 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.873262 25031 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.873564 25031 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:13.874281 25031 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.875494 25031 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.879252 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:13.879805 25619 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.883106 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:13.886030 25031 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.887683 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:13.888777 26326 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.889237 26326 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.890064 26326 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.892122 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.895666 24948 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:13.909585 26410 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:13.926822 26410 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:13.927666 26410 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:13.938385 26410 ts_tablet_manager.cc:1875] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:13.953570 26375 raft_consensus.cc:979] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.954125 26267 raft_consensus.cc:1067] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.957334 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:13.957859 24949 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.958235 24949 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.967476 24949 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.969283 24949 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:13.970139 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.970677 25029 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.970764 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:13.971360 25096 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.971639 25096 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:13.972343 25096 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.976191 25096 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.976716 25029 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.977463 26326 raft_consensus.cc:979] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:13.979753 26307 raft_consensus.cc:1067] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:13.981230 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:13.981909 26307 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.982013 25098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58866
I20240426 13:19:13.982378 26307 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.982532 25098 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.982887 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.983260 26307 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.978345 24853 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:13.989202 24851 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.020531 25029 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:14.039991 24948 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:14.042529 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:14.044605 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:14.046625 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:14.064694 26253 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.068753 24949 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.073583 25619 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.087116 25098 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.127244 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:14.128005 24949 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:14.128394 24949 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:14.129391 24949 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.151338 24949 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:14.150394 25098 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:14.153369 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:14.172520 25030 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:14.164342 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53343363b25040c29f4c212b440dc2cc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:14.157184 26410 ts_tablet_manager.cc:1888] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:14.182410 26410 log.cc:1196] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/764a1a5c07f94909b2fd69a50de4bedd
I20240426 13:19:14.189936 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 764a1a5c07f94909b2fd69a50de4bedd (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:14.183738 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.148401 25096 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:14.193977 24949 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:14.194207 25030 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:14.195852 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:14.196951 26375 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:14.197399 26375 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:14.200141 26375 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.212916 24851 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:14.214227 24851 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:14.220103 24852 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.234455 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 53343363b25040c29f4c212b440dc2cc with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:14.241365 25096 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:14.248195 26307 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:14.275416 26415 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:19:14.345556 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56412.6us
I20240426 13:19:14.359506 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.365670 26326 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:14.372874 25029 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.374367 25615 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.378690 24949 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.382205 26398 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:14.388015 26338 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:14.433320 26415 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:14.434429 26415 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:14.439240 26415 ts_tablet_manager.cc:1875] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:14.891749 26415 ts_tablet_manager.cc:1888] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20240426 13:19:14.892417 26415 log.cc:1196] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/53343363b25040c29f4c212b440dc2cc
I20240426 13:19:14.967813 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:15.270458 26354 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.271214 26354 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.273742 26354 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.468458 26354 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.469058 26354 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.471197 26354 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.480535 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.481216 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:15.482453 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:15.483322 26425 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:15.483938 26425 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.484246 26425 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:15.494688 26354 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.495280 26354 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.497370 26354 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.497632 26429 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.506007 26427 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.498122 26429 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.506772 26427 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.508316 26429 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.515092 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.516163 26424 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.516703 26424 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.518671 26424 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:15.525043 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.13096e+06us
I20240426 13:19:15.529631 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 53343363b25040c29f4c212b440dc2cc (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:15.538185 26434 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.538851 26434 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.539299 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.540841 26434 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.541960 26427 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.542346 26413 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.542860 26413 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.547438 26291 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.548161 26291 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.550289 26291 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.551287 26425 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.545109 26427 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.554354 26427 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.561743 26413 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.562727 26427 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.565833 26427 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.566396 26427 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.568986 26433 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.569541 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:15.570209 25620 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:15.570333 26430 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.569504 26433 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.570796 26430 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.571236 25025 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:15.573113 26430 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.573783 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.575564 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.580507 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 263) took 1187 ms. Trace:
I20240426 13:19:15.580554 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:15.580641 26425 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.581036 24853 rpcz_store.cc:276] 0426 13:19:14.392787 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.577635 (+1184848us) service_pool.cc:225] Handling call
0426 13:19:15.580451 (+  2816us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
W20240426 13:19:15.583715 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s	user 0.007s	sys 0.000s
W20240426 13:19:15.584268 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.186s	user 0.008s	sys 0.000s
I20240426 13:19:15.585088 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 112) took 1189 ms. Trace:
I20240426 13:19:15.585465 24949 rpcz_store.cc:276] 0426 13:19:14.396106 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:14.396233 (+   127us) service_pool.cc:225] Handling call
0426 13:19:14.396540 (+   307us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:14.396854 (+   314us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:14.396880 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:14.397569 (+   689us) log.cc:842] Serialized 26 byte log entry
0426 13:19:15.584682 (+1187113us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:15.584733 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:15.584813 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:15.584862 (+    49us) raft_consensus.cc:1671] finished
0426 13:19:15.584878 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:15.585010 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":636,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":1186716,"threads_started":1,"wal-append.queue_time_us":963,"wal-append.run_cpu_time_us":1710,"wal-append.run_wall_time_us":1006116}
I20240426 13:19:15.589519 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:15.594318 26429 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:15.594858 26429 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:15.594935 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.596204 26429 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.596776 26427 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.600801 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.608286 26424 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.611002 26424 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.611620 26427 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.612133 26427 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.612888 26424 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.616706 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.547477 26431 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:15.629770 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.22126e+06us
I20240426 13:19:15.622061 26427 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:15.634610 23801 debug-util.cc:398] Leaking SignalData structure 0x7b080000ebe0 after lost signal to thread 24921
I20240426 13:19:15.635267 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.636603 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.636330 26433 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.635519 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.639473 26326 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.639853 26424 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.639984 26326 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.640388 26424 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.641924 26326 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.642583 26424 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.643205 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.646008 26424 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.651178 26424 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.653928 26424 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.648002 26419 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.662318 26419 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.664204 26419 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.627264 26431 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.645627 26433 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.667451 26433 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.672493 26428 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.682227 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.699106 26442 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:15.699924 26433 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.707655 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.703922 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.709815 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.673385 26424 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.710443 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:15.717936 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.718549 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:19:15.736932 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104293us
I20240426 13:19:15.737538 26354 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:15.755935 26449 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:15.719039 26375 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.715495 26424 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.684880 26428 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:15.805488 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65140.2us
I20240426 13:19:15.805527 26427 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.825613 26432 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.825781 26435 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.826184 26432 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.827047 26446 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:15.827466 26446 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.824054 26354 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.830096 26452 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.811375 26419 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.830585 26452 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:15.831054 24901 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:15.830958 26419 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.832561 26452 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.832839 26419 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.826239 26435 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.829828 26354 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.834462 26426 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.834937 26426 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.835287 26354 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.836562 26435 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.829363 26446 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.836855 26426 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.837363 26453 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.837450 26432 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.765410 26375 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.847069 26427 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.807062 26425 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.811878 26438 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:15.843904 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aca63e93aec8413b83ec63e853c8f403, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.837899 26453 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.856940 26431 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.858685 26424 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.859329 26430 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.859771 26430 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.861220 26460 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.861563 26430 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.861752 26460 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.863570 26453 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.863819 26460 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.845510 26428 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.847407 26375 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:15.848875 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aca63e93aec8413b83ec63e853c8f403, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.865272 24851 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:15.844723 26450 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.849402 26443 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.849910 26437 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.850489 26427 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.848306 26425 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.765223 26449 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.848372 26438 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.867645 26450 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.896222 26460 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.896970 26460 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.897863 26450 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.837527 26444 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.902901 26431 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:15.914707 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 210641us
I20240426 13:19:15.902743 26444 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.910907 26427 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.927949 26438 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.886384 26291 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.928555 26291 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.929267 26425 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.925945 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.878456 26443 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.931208 25620 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:15.903798 26433 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.878585 26437 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.937477 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:15.932462 26442 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.936070 26467 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.938560 26445 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.938380 26433 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.938851 26444 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.936259 26443 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.936740 26419 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.938683 26467 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:15.936283 26453 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.942349 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.943101 25098 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:15.939101 26445 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.944689 26433 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.927953 26427 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.946328 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "256415e42b6c444387115ab766a5a16c"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:15.947731 24949 raft_consensus.cc:595] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:15.944775 26449 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.948921 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.947290 26453 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.940553 26425 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:15.910717 26431 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.947108 26419 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.948783 26438 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.955171 26453 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.955458 26425 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.955710 26472 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.964699 26425 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:15.959209 26443 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.948204 26326 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.940470 26442 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.962316 26438 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.962756 26446 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.963167 26449 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.963899 25031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:15.964210 26467 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.958997 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:15.948042 26460 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.969746 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.965546 26428 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.950111 26360 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.972685 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.972611 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.959693 26474 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.978917 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:15.964900 26472 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.975852 25031 raft_consensus.cc:595] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:15.977277 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.966099 26413 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.969170 26455 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.969197 26461 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.960402 26450 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.974072 26419 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.979244 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.992192 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:15.981958 25096 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:15.993222 24949 raft_consensus.cc:595] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:15.993520 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.979642 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.969434 26454 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.994038 25097 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:15.982134 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.973819 26326 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.952278 26291 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.983225 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.996017 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.994347 25101 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:16.090940 26459 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.091549 26473 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:15.973160 26443 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.091044 26439 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:16.087427 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55227.7us
I20240426 13:19:15.991226 26456 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.981359 26398 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.092186 26473 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.996827 26453 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.977468 26446 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.983693 26474 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.106148 26466 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.995617 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.990985 26437 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.990867 26458 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.097599 26464 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.978386 26449 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.103332 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.087604 26444 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.004421 26445 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.104581 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.981516 26428 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.105746 26440 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:16.096911 26291 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.981621 26360 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.091640 26459 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.112581 25030 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:15.992774 26413 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.106632 26466 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.004369 26450 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.004235 26454 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.130882 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.209211 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:16.182360 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 461870us
I20240426 13:19:16.185497 25031 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:16.113179 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.135279 25097 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:16.235862 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.992874 26455 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.236904 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.160895 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:16.206471 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 97573.3us
I20240426 13:19:16.246883 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.232635 25619 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:16.247399 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.247972 24948 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:16.249217 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.105332 26467 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.253440 26459 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.253803 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a6a69cdcf2ee4768a20888dd974c6ba7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:16.257696 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a6a69cdcf2ee4768a20888dd974c6ba7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.250002 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.257263 25096 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.258589 26438 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.259881 26455 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.260516 26326 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.202387 26427 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.246338 26432 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.234381 26465 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.261193 26467 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.261348 26432 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.109091 26419 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.105489 26473 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.120368 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.107935 26439 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.203928 26463 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.246584 26431 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.115559 26441 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.004364 26461 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.278574 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.289865 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.109370 26398 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.299141 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.300305 26398 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.261646 26465 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.248083 26444 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.306447 24956 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:16.310504 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.262630 26442 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.247905 26291 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.313750 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.323668 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.316277 24952 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:16.328291 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.329414 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.333431 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.334007 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.120870 26453 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.291430 26474 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:16.334666 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96650us
I20240426 13:19:16.161680 26456 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.314368 26433 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:16.334127 26428 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.340987 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49249.5us
I20240426 13:19:16.263473 26464 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.339979 25619 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.314834 26472 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.341827 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.290488 26419 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.333482 26430 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.333729 26354 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.262876 26440 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.348457 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.356982 26453 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.362130 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.363184 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.366500 24954 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.337430 26443 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.368438 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.368626 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.371480 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.342758 26375 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.372622 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.375186 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.370966 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.364708 26428 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:16.364753 26446 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.392602 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.361917 26354 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.393705 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.397220 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.395138 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.393744 24953 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:16.314491 26463 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.398739 24950 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.374447 26375 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.392357 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:16.399734 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.405882 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.263332 26458 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.406966 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.394857 26428 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.408156 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.414144 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.415606 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.416072 24950 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.409091 24956 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.410122 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.410835 26466 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.420284 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.397878 26472 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.411844 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.411896 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.327656 26441 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.406528 26375 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.400596 26435 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.405830 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.417366 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.398480 26354 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.397110 26467 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.411195 26291 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.421547 24950 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.406252 26449 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.418107 26428 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:16.399333 26454 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.428314 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.429328 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.434381 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:16.435930 26441 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.419900 26456 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.422312 26450 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.440789 26468 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.425401 26439 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.439668 26435 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.440564 26466 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240426 13:19:16.411851 26413 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.446607 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.473444 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.474288 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:16.447849 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 88516.7us
I20240426 13:19:16.475360 26440 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.475448 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:16.449173 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 349644us
I20240426 13:19:16.477144 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.477377 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.449626 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.418622 26461 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.450949 25096 raft_consensus.cc:595] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.451817 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.422780 26472 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.479398 26472 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.453900 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.479809 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.479941 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.460834 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.461081 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.480921 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.431406 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.481443 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.444896 26463 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.447152 26444 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.482601 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.482896 26458 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.465734 26425 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.000s
I20240426 13:19:16.483597 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.483804 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.483482 26425 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.475950 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.487576 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.488314 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.488744 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.486712 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.493366 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:16.509101 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.509905 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:16.764858 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.766618 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:16.511042 26419 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.508951 26485 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.771477 26485 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:16.700515 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d94c0 after lost signal to thread 23799
W20240426 13:19:16.772929 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9ba0 after lost signal to thread 24888
W20240426 13:19:16.774230 25645 debug-util.cc:398] Leaking SignalData structure 0x7b080034dd00 after lost signal to thread 24980
W20240426 13:19:16.775367 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800356d20 after lost signal to thread 25055
I20240426 13:19:16.771901 26485 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.776072 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.465816 26468 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.748450 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.779852 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.780915 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.748932 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.492317 26413 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.781854 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.750388 26435 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.495627 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:16.783653 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 288083us
I20240426 13:19:16.783516 26432 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.784497 26432 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.784931 26432 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.785073 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.785427 26432 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.786190 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.786615 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.750859 26375 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.753345 26472 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.753643 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.804955 26465 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.808303 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800216940 after lost signal to thread 25129
W20240426 13:19:16.809937 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800370e60 after lost signal to thread 25650
I20240426 13:19:16.810477 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.812404 26437 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.813161 26437 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.813728 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.814922 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.815155 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.816154 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.816357 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817035 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817385 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817649 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.818847 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.820305 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.821825 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.787228 26468 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.823724 26460 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:16.824235 26460 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.824676 26460 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.479296 26480 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.757491 25102 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:16.829458 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.758311 26473 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.758788 26425 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.761469 26450 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.835530 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.801801 26427 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.761183 26428 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.763000 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.764180 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.764700 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.770661 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:16.778592 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.872905 24954 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.782872 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.508905 26494 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:16.811931 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.832010 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:16.877969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46039.5us
I20240426 13:19:16.879107 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:16.879559 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67725.7us
I20240426 13:19:16.834775 26469 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.834566 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.836122 26450 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.762151 26464 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.842867 26466 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.850464 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.854537 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.892311 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.856129 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.858238 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.838786 26472 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.862917 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.881423 24948 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:16.894747 24948 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:16.882288 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.895089 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.895432 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:16.865756 26375 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.886569 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.888536 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.883229 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:16.906075 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.903784 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.906947 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:16.908078 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.897212 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.898401 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.896526 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.882165 26435 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.882757 26438 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.913801 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.892657 26450 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.891230 26430 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.913856 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.916412 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:16.916867 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.917224 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.890127 26425 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.908867 24948 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.912483 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.890549 26480 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.920377 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:16.912830 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.974145 24948 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.877063 26468 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.913964 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.972946 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.976256 26468 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.976326 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.977479 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:16.905113 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 142190us
I20240426 13:19:16.978404 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.979317 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.980258 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.980718 26360 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.981403 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:16.981685 26473 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.982117 25624 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:16.982542 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.983600 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.918448 26375 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.983856 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.984210 26375 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.928758 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.984673 26375 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.984968 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.928656 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.985926 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:16.983767 26453 consensus_queue.cc:1159] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:16.980346 26428 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.999433 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:16.991083 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.001183 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.002596 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:16.992976 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80557us
W20240426 13:19:16.993749 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77398.9us
I20240426 13:19:16.918578 26433 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.011823 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.915868 26472 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.013760 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.928555 26438 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.038810 26438 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.041923 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.039605 25029 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:17.044188 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.048939 25618 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.988607 26425 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.053859 26425 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.058612 26427 raft_consensus.cc:979] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:17.059049 26427 raft_consensus.cc:1067] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
W20240426 13:19:17.060814 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 551154b9c3494e4da288affc4b88c361, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.061161 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 551154b9c3494e4da288affc4b88c361, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.014497 26494 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.010926 26458 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240426 13:19:17.014377 26413 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.014585 26481 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.014137 26433 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.069651 26433 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:17.001005 26354 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.006s
I20240426 13:19:17.072387 26354 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.074695 26354 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.011358 26468 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.011091 26426 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.077697 26426 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.078001 26426 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.924702 26488 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:17.081876 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45051.5us
I20240426 13:19:16.924386 26469 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:17.092882 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 163520us
I20240426 13:19:16.903213 26466 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.081936 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68277.6us
I20240426 13:19:17.095870 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.031158 24948 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:17.094431 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.100730 26458 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:17.101140 26458 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:17.101919 26458 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.103130 26458 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:17.014238 26436 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:17.104310 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98782.7us
I20240426 13:19:16.973793 26430 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.108985 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.109261 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.105674 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.924073 26419 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.177s
I20240426 13:19:17.108414 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.120122 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.121558 24952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:17.122031 24952 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:17.122426 24952 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.109354 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:17.106333 26439 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240426 13:19:17.110515 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:17.110603 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 213447us
I20240426 13:19:17.128854 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:16.978379 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.113297 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.107766 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.118274 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.119163 24954 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:17.126780 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.100142 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.130821 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.132264 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.132706 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.118834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:17.134603 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.095237 26469 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.126397 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.107159 26456 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.146255 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.146761 26375 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.147231 26497 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:17.147457 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.147775 26497 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.148609 25615 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:17.149363 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:17.149608 26497 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.150131 26448 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:17.140090 26461 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:17.117470 26430 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.126488 26435 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.150420 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.134604 26458 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.175031 26434 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.139854 26469 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.189224 26505 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240426 13:19:17.189996 26466 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.129096 26488 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20240426 13:19:17.196035 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 90447.6us
I20240426 13:19:17.152212 26436 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.148631 26375 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.202440 26505 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.180552 26483 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.206719 24952 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.142026 26432 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.202872 26505 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:17.214247 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87922.1us
I20240426 13:19:17.214965 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.215560 24952 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:17.188673 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 210378us
I20240426 13:19:17.223440 25104 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.223513 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.226503 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.227715 26509 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.228596 26509 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:17.219833 26375 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.208992 26488 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.234488 26432 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.235275 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117063us
I20240426 13:19:17.232719 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.241291 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.209900 26507 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.244784 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.248278 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.249715 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.249682 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.244323 26360 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.252135 25100 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.239681 26375 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.253190 25617 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:17.253990 26375 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.248201 26465 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.252386 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.251722 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.255875 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.225039 26444 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:17.257992 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.258119 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.239636 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.258594 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.259898 25103 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:17.258008 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.263319 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.265476 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:17.235589 26398 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 13:19:17.279063 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.279392 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.274847 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.259406 26375 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.282195 26433 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.284293 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.285753 26473 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.277915 26438 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:17.286695 26438 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.287147 26438 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:17.286808 26425 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.288882 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.286162 26473 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.289412 25098 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.288890 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.290622 26432 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.290892 26433 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.291690 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.273185 26446 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.293603 26508 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:17.256759 26507 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.297259 26497 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.297677 26497 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.297852 26291 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.258039 26503 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.275262 26424 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:17.291435 26473 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.277805 26436 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240426 13:19:17.305843 26436 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:17.306149 26436 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.292361 26428 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.319291 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.295185 26437 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:17.256865 26505 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.298504 26507 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.299042 26433 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.337107 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.295209 26425 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.343680 26460 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.518s	user 0.008s	sys 0.000s
I20240426 13:19:17.347292 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.355321 24851 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 277) took 1557 ms. Trace:
W20240426 13:19:17.301148 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69495.6us
I20240426 13:19:17.357501 26450 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.368491 26460 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.306830 26436 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.307533 26497 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.373431 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51319.4us
W20240426 13:19:17.374027 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48990.7us
I20240426 13:19:17.381424 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.348400 26469 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.302421 26424 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.383349 26469 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.323889 26437 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:17.369135 24851 rpcz_store.cc:276] 0426 13:19:15.792598 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.792754 (+   156us) service_pool.cc:225] Handling call
0426 13:19:17.349686 (+1556932us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:15.982821 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:16.190808 (+207987us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:16.190838 (+    30us) write_op.cc:427] Acquired schema lock
0426 13:19:16.190855 (+    17us) tablet.cc:660] Decoding operations
0426 13:19:16.249948 (+ 59093us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:16.262359 (+ 12411us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:16.263915 (+  1556us) tablet.cc:683] Acquiring locks for 1 operations
0426 13:19:16.302170 (+ 38255us) tablet.cc:699] Row locks acquired
0426 13:19:16.306309 (+  4139us) write_op.cc:260] PREPARE: finished
0426 13:19:16.329881 (+ 23572us) write_op.cc:270] Start()
0426 13:19:16.335942 (+  6061us) write_op.cc:275] Timestamp: P: 1714137556327441 usec, L: 0
0426 13:19:16.337535 (+  1593us) op_driver.cc:331] REPLICATION: starting
0426 13:19:16.408638 (+ 71103us) log.cc:842] Serialized 530 byte log entry
0426 13:19:16.919686 (+511048us) op_driver.cc:444] REPLICATION: finished
0426 13:19:17.190443 (+270757us) write_op.cc:300] APPLY: starting
0426 13:19:17.190551 (+   108us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:17.190679 (+   128us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:17.190814 (+   135us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:17.191065 (+   251us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:17.191276 (+   211us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 13:19:17.191341 (+    65us) write_op.cc:311] APPLY: finished
0426 13:19:17.191611 (+   270us) log.cc:842] Serialized 23 byte log entry
0426 13:19:17.191803 (+   192us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:17.191903 (+   100us) write_op.cc:446] Released schema lock
0426 13:19:17.191955 (+    52us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:17.192301 (+   346us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:17.192337 (+    36us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":155184,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":267508,"apply.run_cpu_time_us":2116,"apply.run_wall_time_us":2114,"num_ops":1,"prepare.queue_time_us":105066,"prepare.run_cpu_time_us":68686,"prepare.run_wall_time_us":771389,"raft.queue_time_us":171174,"raft.run_cpu_time_us":26845,"raft.run_wall_time_us":197623,"replication_time_us":582061,"thread_start_us":661361,"threads_started":4,"wal-append.queue_time_us":329174}]]}
I20240426 13:19:17.383944 26437 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.384776 25100 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:17.397647 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.399633 25099 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:17.404629 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45056.9us
I20240426 13:19:17.383368 26424 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.409420 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.383788 26426 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.430330 26512 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:17.434303 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.412673 26436 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.407053 26509 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.415992 26291 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.432258 26510 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:17.445691 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:17.436985 26485 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.660s	user 0.008s	sys 0.003s
I20240426 13:19:17.402534 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:17.454028 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 210746us
I20240426 13:19:17.455508 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.456732 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.458948 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.464465 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.465747 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.467617 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.449791 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:17.471467 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.442623 26512 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.469656 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:17.474293 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60923.5us
I20240426 13:19:17.462010 26444 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.475090 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.465384 26458 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.467073 26467 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.467483 26488 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:17.476231 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.467027 26428 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:17.459728 26439 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.498661 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:17.486856 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49305.6us
I20240426 13:19:17.485148 26426 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.477026 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76487.2us
I20240426 13:19:17.474572 26465 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.474925 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1102) took 1358 ms. Trace:
I20240426 13:19:17.525774 25026 rpcz_store.cc:276] 0426 13:19:16.111189 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.112348 (+  1159us) service_pool.cc:225] Handling call
0426 13:19:16.116431 (+  4083us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:16.400326 (+283895us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:19:16.760077 (+359751us) log.cc:842] Serialized 14 byte log entry
0426 13:19:16.961562 (+201485us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:16.997671 (+ 36109us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:17.001163 (+  3492us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:17.456623 (+455460us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b54002625d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:17.458823 (+  2200us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:17.469611 (+ 10788us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16019,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":6381,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":11958,"spinlock_wait_cycles":639824128,"thread_start_us":438399,"threads_started":2,"wal-append.queue_time_us":302272}
I20240426 13:19:17.456786 26485 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.530731 26485 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.517704 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.514918 26428 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.532497 26428 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:17.532442 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.477762 26433 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.520092 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:17.521158 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.521544 26460 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.483351 26513 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.534065 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.534772 25619 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.479593 26456 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.552752 26456 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.553010 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.542781 26454 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.553844 26454 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:17.554067 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.554266 26430 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.542871 26465 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.543756 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.556746 26448 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.557080 26448 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.544526 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.557871 26448 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.559330 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.560544 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.561291 26442 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.561869 26442 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.562398 26442 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.565632 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.569525 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.570575 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.571203 25028 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.571322 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.545527 26428 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.503571 26488 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:17.593874 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.572454 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.544620 26438 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.555246 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:18.563351 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.00811e+06us
I20240426 13:19:18.564013 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.566843 26505 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.580583 25097 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.583623 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.483597 26291 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20240426 13:19:18.568655 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 985068us
I20240426 13:19:18.569912 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.594126 25103 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.570734 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 161) took 1014 ms. Trace:
I20240426 13:19:18.571125 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1186) took 1001 ms. Trace:
I20240426 13:19:18.571107 25104 rpcz_store.cc:276] 0426 13:19:17.556746 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.569099 (+ 12353us) service_pool.cc:225] Handling call
0426 13:19:18.569804 (+1000705us) spinlock_profiling.cc:243] Waited 876 us on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:18.570679 (+   875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2066560}
I20240426 13:19:17.555912 26465 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.571579 25103 rpcz_store.cc:276] 0426 13:19:17.569607 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570024 (+   417us) service_pool.cc:225] Handling call
0426 13:19:17.593963 (+ 23939us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.570899 (+976936us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.571074 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8960}
I20240426 13:19:17.521639 26450 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.522243 26458 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:18.572222 26463 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.597247 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:18.573045 26354 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.573519 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1471) took 1004 ms. Trace:
I20240426 13:19:18.573580 26354 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.572487 26465 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.572530 26450 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.601547 26438 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.595917 26488 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.603268 26428 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.558987 26448 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.614850 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.621023 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.631271 25028 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:17.756635 26481 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.598248 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.665832 26473 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.786480 26437 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.797433 26430 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.885805 26515 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.961025 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1172) took 1131 ms. Trace:
I20240426 13:19:17.985703 26485 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.059396 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 237) took 1174 ms. Trace:
I20240426 13:19:18.460880 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1445) took 1007 ms. Trace:
I20240426 13:19:18.472329 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1164) took 1667 ms. Trace:
I20240426 13:19:18.483134 26413 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.486124 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.490020 26483 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.496923 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 383) took 1196 ms. Trace:
I20240426 13:19:18.528537 26467 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.612673 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.533489 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.798797 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 112) took 1194 ms. Trace:
I20240426 13:19:18.799146 25022 rpcz_store.cc:276] 0426 13:19:17.604463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.605631 (+  1168us) service_pool.cc:225] Handling call
0426 13:19:18.798720 (+1193089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.800145 25617 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.800253 26473 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.549057 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.573910 25027 rpcz_store.cc:276] 0426 13:19:17.568820 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.568977 (+   157us) service_pool.cc:225] Handling call
0426 13:19:18.573443 (+1004466us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:17.570504 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.846168 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1476) took 1276 ms. Trace:
I20240426 13:19:18.846508 25030 rpcz_store.cc:276] 0426 13:19:17.570183 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570293 (+   110us) service_pool.cc:225] Handling call
0426 13:19:18.846102 (+1275809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:17.707365 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:18.572571 26431 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.530205 26441 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.874193 26441 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.620569 26481 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.617689 26483 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.897927 26481 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.898468 26483 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.627812 24956 rpcz_store.cc:276] 0426 13:19:17.453283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.453467 (+   184us) service_pool.cc:225] Handling call
0426 13:19:17.468659 (+ 15192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.482906 (+ 14247us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.482941 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:17.482985 (+    44us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:17.483012 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.460633 (+977621us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.460697 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.460815 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32395648}
I20240426 13:19:18.636610 26485 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.900429 26485 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:18.642448 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.039s	user 0.031s	sys 0.075s
W20240426 13:19:18.901043 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.297s	user 0.031s	sys 0.075s
W20240426 13:19:18.642299 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78341.7us
I20240426 13:19:18.651263 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:18.901803 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1444) took 1457 ms. Trace:
I20240426 13:19:18.575897 26450 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.902112 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.902732 26450 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.903332 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.904156 26450 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.904475 26505 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.905072 26505 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.902153 24950 rpcz_store.cc:276] 0426 13:19:17.444222 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.444387 (+   165us) service_pool.cc:225] Handling call
0426 13:19:17.444792 (+   405us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:17.445187 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.445210 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:17.603170 (+157960us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:19:17.603495 (+   325us) log.cc:842] Serialized 26 byte log entry
0426 13:19:18.901386 (+1297891us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:18.901419 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.901479 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:18.901519 (+    40us) raft_consensus.cc:1671] finished
0426 13:19:18.901532 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.901721 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":330,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4943,"spinlock_wait_cycles":3733760,"thread_start_us":1297543,"threads_started":1,"wal-append.queue_time_us":313100}
I20240426 13:19:18.687144 26493 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.908298 26493 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.909437 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.910204 26465 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.910305 26493 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.687587 26527 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.690323 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.911042 26527 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:18.911471 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 221205us
I20240426 13:19:18.912336 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.913434 26527 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.913543 26507 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.914029 26527 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:18.713445 25099 rpcz_store.cc:276] 0426 13:19:16.804663 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.032091 (+227428us) service_pool.cc:225] Handling call
0426 13:19:17.036580 (+  4489us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.400676 (+364096us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.472016 (+1071340us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.472247 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832620800}
I20240426 13:19:18.715318 25105 rpcz_store.cc:276] 0426 13:19:16.829595 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.039456 (+209861us) service_pool.cc:225] Handling call
0426 13:19:17.039698 (+   242us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.292815 (+253117us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:17.960702 (+667887us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:17.960956 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572681472}
W20240426 13:19:18.717132 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.12813e+06us
I20240426 13:19:18.916728 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.918370 26505 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.581255 26436 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20240426 13:19:18.920004 26436 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.920424 26436 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.921341 26436 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.922698 26436 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.732303 25619 rpcz_store.cc:276] 0426 13:19:16.885340 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.885454 (+   114us) service_pool.cc:225] Handling call
0426 13:19:16.885665 (+   211us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.552475 (+666810us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.059184 (+506709us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.059325 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1492279168}
I20240426 13:19:18.928051 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.738065 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.927145 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.929646 26493 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.929805 26505 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.930223 26493 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.932288 26493 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.933131 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:18.933753 25030 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:18.935518 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.936637 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.937220 24955 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:18.937336 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.938091 25105 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:18.939240 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.940337 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.940936 24954 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.943445 26481 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:18.738575 26428 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.575937 26354 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.944607 26481 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:18.945792 26446 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:18.946127 26446 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.946870 26446 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.743971 26413 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.772462 26515 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.745716 25618 rpcz_store.cc:276] 0426 13:19:17.300171 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.301095 (+   924us) service_pool.cc:225] Handling call
0426 13:19:17.344232 (+ 43137us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:17.414156 (+ 69924us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.416288 (+  2132us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:17.431932 (+ 15644us) log.cc:842] Serialized 26 byte log entry
0426 13:19:17.562800 (+130868us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b18002cbf60. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:17.612657 (+ 49857us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:17.612709 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.496545 (+883836us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.496606 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:18.496671 (+    65us) raft_consensus.cc:1671] finished
0426 13:19:18.496688 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.496860 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2224,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17314,"spinlock_wait_cycles":107273984,"thread_start_us":49662,"threads_started":1,"wal-append.queue_time_us":184615,"wal-append.run_cpu_time_us":1639,"wal-append.run_wall_time_us":1001692}
I20240426 13:19:18.774853 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.779492 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.793933 26488 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.948004 26446 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:18.949440 26413 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.949440 26515 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.809155 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1475) took 1239 ms. Trace:
I20240426 13:19:18.952210 26488 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.953300 25028 rpcz_store.cc:276] 0426 13:19:17.569908 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570395 (+   487us) service_pool.cc:225] Handling call
0426 13:19:18.809083 (+1238688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.949819 26438 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.816146 26437 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.817416 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1470) took 1249 ms. Trace:
I20240426 13:19:18.953943 26438 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.954275 25026 rpcz_store.cc:276] 0426 13:19:17.568506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.594458 (+ 25952us) service_pool.cc:225] Handling call
0426 13:19:18.817359 (+1222901us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.954800 26438 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.956645 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:18.957496 26428 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:18.957852 26428 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.958192 26428 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:18.955845 26438 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.824313 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:18.822839 26503 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:18.962991 25104 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:18.963276 26503 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.804010 25029 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:18.828392 26448 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:18.829723 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.20869e+06us
I20240426 13:19:18.963689 26503 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:18.964558 26503 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.965932 26503 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.819520 26467 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.834676 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.969518 26467 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.970978 26437 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.836493 24854 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 264) took 2889 ms. Trace:
I20240426 13:19:18.972604 24854 rpcz_store.cc:276] 0426 13:19:15.946901 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.947044 (+   143us) service_pool.cc:225] Handling call
0426 13:19:18.836407 (+2889363us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20240426 13:19:18.974272 26449 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.974465 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.974833 26449 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.971489 26437 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.841609 26497 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.977178 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.847998 25616 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:18.978386 26437 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.833981 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1173) took 2004 ms. Trace:
I20240426 13:19:18.871246 26526 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:18.979012 25097 rpcz_store.cc:276] 0426 13:19:16.829940 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.043162 (+213222us) service_pool.cc:225] Handling call
0426 13:19:17.043416 (+   254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.567852 (+1524436us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.833785 (+265933us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.833931 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1230400768}
I20240426 13:19:18.876482 26441 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.885509 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.980608 25623 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:18.889360 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 241) took 1459 ms. Trace:
I20240426 13:19:18.969595 26472 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.981449 25617 rpcz_store.cc:276] 0426 13:19:17.446285 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.451101 (+  4816us) service_pool.cc:225] Handling call
0426 13:19:17.456400 (+  5299us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.800609 (+1344209us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.889084 (+ 88475us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540063f454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.889300 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3089833216}
I20240426 13:19:18.981709 26472 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.917503 26465 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.982787 26465 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.983772 25615 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:18.839756 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 270) took 1977 ms. Trace:
I20240426 13:19:18.978587 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 243) took 1357 ms. Trace:
I20240426 13:19:18.867452 26524 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:18.830838 26291 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.979578 26526 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.983959 26472 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.999584 26448 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.986305 26494 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.000578 26494 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.001219 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.001782 26448 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.002023 25617 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:19.002621 26494 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.987095 26413 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240426 13:19:19.003861 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.003968 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.004191 25616 rpcz_store.cc:276] 0426 13:19:17.620815 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.628142 (+  7327us) service_pool.cc:225] Handling call
0426 13:19:18.978509 (+1350367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:19.004539 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1473) took 1435 ms. Trace:
I20240426 13:19:19.002341 24852 rpcz_store.cc:276] 0426 13:19:16.862058 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.862236 (+   178us) service_pool.cc:225] Handling call
0426 13:19:18.839687 (+1977451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1265328,"reported_tablets":1}
I20240426 13:19:19.004984 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.004887 25029 rpcz_store.cc:276] 0426 13:19:17.569380 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.571093 (+  1713us) service_pool.cc:225] Handling call
0426 13:19:19.004392 (+1433299us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:19.004485 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2793008896}
I20240426 13:19:19.007638 26494 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.007993 26494 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.008373 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.009035 25623 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:19.008808 26494 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.011411 26494 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.001353 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.012835 26524 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.986801 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.014982 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.015331 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.015702 25099 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.015905 25104 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:19.016896 26526 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.017346 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.018108 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.996593 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.019830 24950 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.020170 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.020745 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:19.022006 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.022580 25100 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:18.987422 26465 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.991726 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.025368 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.026283 26536 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:19.026758 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.027310 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:19.030216 26481 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:19.010401 26483 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.030555 26494 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.030848 26481 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.031728 26481 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.033161 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.976771 26449 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:19.009819 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82770.5us
W20240426 13:19:18.997402 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82511us
I20240426 13:19:19.037745 24954 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:19.038326 24953 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.040207 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.030126 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.042171 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.042912 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.044468 26483 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.044759 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.044859 26413 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.060264 26483 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.060803 26413 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.061587 26413 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.062425 26413 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.062374 26483 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.063311 26507 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.064890 26446 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:19.066100 26446 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.066386 26446 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:19.043562 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44786.9us
I20240426 13:19:19.046509 24955 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:19.049086 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.990386 26441 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.058406 26526 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.070063 24948 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:19.051533 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.052557 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.054852 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.043938 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.058880 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.073822 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.017541 26524 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.069761 26441 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.050537 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.067310 26428 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.078640 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:19.081264 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45995.2us
I20240426 13:19:19.109588 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.077271 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.079404 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.111465 26449 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.113467 26494 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.113646 26493 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.092458 26473 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.113770 26449 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.114038 26494 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.092839 26472 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.115056 26472 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.094651 26425 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.115981 26449 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.117753 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.118530 24952 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:19.119067 26413 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:19.117830 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.119472 26413 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.119982 26413 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.094496 26483 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:19:19.120679 26448 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.121176 26448 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.122067 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.123036 26448 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.092800 26438 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.123592 26448 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:19.119415 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.124212 26291 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:19.124558 26291 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.125329 26291 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.113055 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.067122 26446 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.130990 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.093364 26507 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.112864 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82813.9us
I20240426 13:19:19.146307 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.147261 26473 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:19.147720 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.148772 26473 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.070075 26526 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.126564 26291 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:19.128176 26515 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:19.160409 26515 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.160876 26515 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.156664 26431 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.157380 26441 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.115964 26494 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.126752 26448 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.165900 26448 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.126793 26538 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.168275 26538 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.168013 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.159585 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.158464 26526 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.172724 26472 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.140480 26425 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.138841 26488 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.174188 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:19.175894 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55650.7us
I20240426 13:19:19.176504 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.141993 26527 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:19.175540 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44618.1us
I20240426 13:19:19.178601 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.123643 26438 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.149216 26473 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:19.179793 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.173642 26446 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.191073 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:19.185743 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76211us
I20240426 13:19:19.185657 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.188225 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.188490 25100 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.188424 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.193269 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.193434 25023 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.194718 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.195170 25102 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.158430 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.190670 26450 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.200933 26536 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:19.202623 26536 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:19.203809 26536 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.204686 26538 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.216674 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.217293 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.207111 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.180989 26526 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.219537 25100 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.181326 26441 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.241106 25024 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.221185 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.257092 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.228760 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.283927 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.220803 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.247447 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.284307 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.285084 25101 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.284137 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.217875 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.245813 25097 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.287312 26465 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.288317 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.288658 26428 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.288930 25022 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.295531 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.296311 25027 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.182179 26527 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.177809 26488 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.192814 26431 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.305002 26431 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.305470 26488 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.306844 26431 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.183537 26425 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.312503 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.313506 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.314898 25023 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.218456 26450 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.323168 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.315562 25022 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.316262 25029 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.250175 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.325515 25099 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.327072 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.328389 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44547.5us
W20240426 13:19:19.328518 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 110741us
I20240426 13:19:19.322232 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.246603 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.212255 26481 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.330715 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 101106us
I20240426 13:19:19.320915 25025 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.334291 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.205827 26467 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.354844 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.359875 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.360775 24950 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:19.241699 26493 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.308975 26505 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.322867 26450 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.374256 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.371011 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.371625 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57812.6us
I20240426 13:19:19.372457 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.335901 26508 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.408480 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.402361 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1469) took 1852 ms (client timeout 1537 ms). Trace:
I20240426 13:19:19.343171 26436 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:19.410604 25025 rpcz_store.cc:269] 0426 13:19:17.550181 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.550382 (+   201us) service_pool.cc:225] Handling call
0426 13:19:19.402176 (+1851794us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:19.402287 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3986326272}
W20240426 13:19:19.419317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59267.9us
I20240426 13:19:19.205691 26473 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.402139 26450 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.344024 26448 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:19.421169 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72332.9us
I20240426 13:19:19.360072 26524 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240426 13:19:19.423578 26493 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.365658 26467 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.401664 26465 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240426 13:19:19.401265 26526 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 13:19:19.422660 26494 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.421717 26527 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.442624 26483 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.462263 25027 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.427317 26543 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.415366 26508 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.424117 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.400229 26425 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.458686 26438 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.462755 26494 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.470755 26413 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.426340 26515 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.472002 26494 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.478374 26413 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.479247 26485 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:19.479318 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66598.7us
I20240426 13:19:19.479751 26485 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.480194 26485 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.484283 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.469494 26543 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.487941 26555 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.496085 26555 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.496393 26515 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.496845 26555 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.494740 26465 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.492151 26543 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.498083 26465 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.498075 26555 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.497970 26493 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.501024 26465 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.503549 26465 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.496507 26467 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.504961 26485 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:19.534822 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84515.2us
W20240426 13:19:19.465740 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 272530us
I20240426 13:19:19.540971 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.544848 25096 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.544991 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.549337 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.552716 25028 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.553325 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.555758 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.558239 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:19.559558 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.562268 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.558439 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.563160 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.566744 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.563171 25104 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.571451 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.574159 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:19.575065 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 127572us
I20240426 13:19:19.576372 26561 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.576550 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.579864 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.581532 26562 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.581956 26562 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.582334 26562 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.577930 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.577951 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.577181 26561 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.593231 26561 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.595382 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.596344 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.598381 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:19.600562 26560 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20240426 13:19:19.602527 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.603775 26536 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.608235 26467 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.608769 26467 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.610400 26467 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.611256 26413 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.611665 26485 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.612418 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.613073 26526 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.612584 26505 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20240426 13:19:19.621208 26505 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.621611 26505 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.622675 26505 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.626391 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:19.627908 24977 debug-util.cc:398] Leaking SignalData structure 0x7b0800200240 after lost signal to thread 23799
I20240426 13:19:19.628566 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:19.629299 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3aa0 after lost signal to thread 24888
I20240426 13:19:19.628931 26505 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.629738 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf480 after lost signal to thread 24901
W20240426 13:19:19.632093 24977 debug-util.cc:398] Leaking SignalData structure 0x7b080026aba0 after lost signal to thread 24980
W20240426 13:19:19.628628 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50835.8us
I20240426 13:19:19.614413 26446 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.634732 26436 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.633929 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08003229a0 after lost signal to thread 25055
I20240426 13:19:19.634851 26446 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.637471 26446 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.611989 26413 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.638211 26413 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.638567 26413 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:19.639757 24977 debug-util.cc:398] Leaking SignalData structure 0x7b0800165aa0 after lost signal to thread 25129
I20240426 13:19:19.641722 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:19.613536 26438 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:19.645716 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08003092a0 after lost signal to thread 25650
I20240426 13:19:19.645341 26438 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.648687 25104 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:19.650363 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61766.2us
I20240426 13:19:19.651108 26441 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.634240 26431 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.656574 26441 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.613003 26448 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.634550 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.649657 26438 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.669628 26524 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.673079 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.674325 26503 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.674532 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 85009.7us
I20240426 13:19:19.675444 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:19.686390 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52020.8us
I20240426 13:19:19.689433 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.690434 25096 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.690981 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.692070 26508 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.692330 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:19.658227 26441 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.689522 26450 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.716454 25030 raft_consensus.cc:2354] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.700155 26354 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:19.700253 26291 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.008s
I20240426 13:19:19.701011 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.701928 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.704349 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.700387 26481 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.723750 25026 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 3. Ops: []
I20240426 13:19:19.724314 26450 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.726361 26508 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:19.728595 26508 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.729068 26413 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.729683 26508 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.731091 26413 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:19.733564 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58293.6us
I20240426 13:19:19.734472 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.735364 26431 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.735908 26431 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.736856 26441 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.736879 26431 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.740727 26551 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:19.741814 26551 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.742280 26551 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.743889 26483 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.743891 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.743325 26551 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.739605 25070 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.748478 25070 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:19.725802 26562 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:19.749315 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 499552c0877e46c5af41389ff1be3152, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:19.749677 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 499552c0877e46c5af41389ff1be3152, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.749524 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.750113 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.752643 26562 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.753530 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.753993 25615 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.756378 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.758486 26551 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.764137 25070 leader_election.cc:398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.764485 25070 leader_election.cc:401] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:19.765045 26535 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.765619 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.768110 26526 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.772651 25615 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:19.773852 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:19.774669 26465 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.771804 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.776190 26465 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.776794 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.777375 26465 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.775216 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.786247 26526 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.786592 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.786617 26526 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.787958 26562 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:19.780349 26483 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.788228 26526 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.795543 26553 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.796062 26553 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.797153 26553 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.800993 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.803051 25615 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:19.803229 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.804591 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.791486 26431 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.808741 26553 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.810009 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.810057 24951 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.812777 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:19.813591 26535 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.813943 26535 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.814505 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.797631 26467 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.825574 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.816529 26483 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.818634 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.819788 24952 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:19.820807 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.828261 25623 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:19.827941 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.814244 26535 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.834883 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.835726 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.836930 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.838371 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.838537 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.839246 25027 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.846905 26524 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.847399 26524 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.850158 26505 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.858860 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59992.7us
I20240426 13:19:19.859416 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.860756 26354 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:19.861809 26354 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.862124 26354 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.863026 26354 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.867336 26354 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.869266 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:19.865917 24956 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:19.865861 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.877159 26526 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.878633 26483 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.878579 24852 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.848321 26524 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.884912 26524 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.886170 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86712.5us
I20240426 13:19:19.890987 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.896826 26424 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.897449 26424 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:19.898126 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.899009 26568 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.900027 26568 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.900238 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.901526 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.903528 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.905781 26505 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.909471 26515 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.910112 26483 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.909992 26515 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.911266 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.911898 26515 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.915319 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:19.915059 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.918462 24955 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.904175 26526 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.920953 26431 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.922878 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:19.923151 26535 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.923741 26515 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.923449 26467 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.924188 26515 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.924693 26515 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.925408 26535 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.930789 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.931380 24955 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.924505 26467 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240426 13:19:19.917595 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.936069 26483 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.936318 26551 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.936818 26551 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.938604 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.939316 26507 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.939443 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.939949 26291 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.940941 26507 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.941339 26507 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.937728 26551 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.941687 26507 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.939339 24955 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:19.943248 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.952054 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.942174 26483 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240426 13:19:19.942512 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.943007 26551 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.956840 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.958097 26291 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.958658 26291 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.947155 26515 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.935088 24852 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.959899 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:19.961474 26535 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.961519 26467 raft_consensus.cc:2741] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 13:19:19.962121 26535 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.962433 26467 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.962982 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.963816 26553 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.963933 26467 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.952390 24854 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.964231 26553 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.965046 24854 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.968492 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.969272 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.970310 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.970877 25098 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.968328 26291 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240426 13:19:19.971241 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.965154 26553 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.972380 26568 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.972797 26553 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.973037 26508 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.979173 25098 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:19.981743 26527 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.983498 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.961846 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.986625 26524 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:19.990386 26467 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:19.990842 26526 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.997874 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.998481 24948 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.963400 26535 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.010489 24954 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:20.010380 26503 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.011957 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:20.012092 26563 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:20.013480 26563 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:20.017423 26467 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.019491 26467 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.020666 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.022118 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.024695 26515 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:20.025394 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.026355 25022 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.025497 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.032820 24851 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:20.037540 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:20.039680 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.041642 26551 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.999523 26524 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.045336 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.058646 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:20.069063 25028 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.081821 26515 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.070531 26431 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:20.090581 24954 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:20.104454 26424 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:20.105599 26424 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.106009 26424 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:20.106810 26424 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.107826 24854 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:20.111974 26424 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.134109 25615 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:20.111192 26431 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:20.141036 25022 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.145386 26467 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:20.150367 26524 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:20.152115 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:20.152433 26291 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.153021 25615 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:20.152972 26291 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.146530 26507 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.154832 26291 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.155176 26507 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:20.155767 25098 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:20.156646 26575 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20240426 13:19:20.157013 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.157388 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.157948 25100 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.160961 24954 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.159461 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.168903 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.170178 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.173173 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.172477 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.177141 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:20.143674 26508 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240426 13:19:20.167374 26527 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:20.170578 26483 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.165004 26524 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240426 13:19:20.207391 24956 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.210143 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.192538 26505 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.202963 26483 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.175978 26535 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240426 13:19:20.215600 26505 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.216136 26561 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:20.224570 26483 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.179175 26431 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.248759 24956 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:20.237828 26505 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:20.280753 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146499us
I20240426 13:19:20.294382 26553 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.306869 26553 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.308043 26553 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.309854 26553 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:20.295425 26579 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20240426 13:19:20.300941 26551 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:20.261385 26561 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
W20240426 13:19:20.328013 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 125231us
I20240426 13:19:20.330494 25100 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:20.331399 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 111849us
I20240426 13:19:20.333686 25023 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:20.335566 26481 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:20.338186 25023 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.347865 25098 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:20.359828 26576 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20240426 13:19:20.368362 26562 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.389679 26291 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.390396 26291 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.416443 25623 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:20.427742 26291 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.429174 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.431586 25099 raft_consensus.cc:595] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.432770 24956 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.432708 26467 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:20.435047 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.435431 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.435748 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.436143 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.438377 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.395606 26561 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:19:20.444800 26483 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.449611 26438 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.450297 26413 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.453847 26515 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.486827 25098 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:20.492362 25028 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:20.495779 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 154215us
I20240426 13:19:20.502036 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.484156 26413 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.502667 25096 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:20.489063 26467 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.515185 26413 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.494544 26526 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240426 13:19:20.489972 26438 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:20.539752 26597 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20240426 13:19:20.541173 26505 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.544670 26505 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:20.552467 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.554070 26483 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.556056 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.556715 25098 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.564267 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.564970 26465 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.565160 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.565513 26465 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.567449 26465 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.569912 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:20.570755 25617 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:20.532100 26467 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.578382 26467 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.586520 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.587579 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.588272 25031 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.592602 26438 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.593930 25025 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.597805 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.598639 26553 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.599004 26553 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.599336 26553 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:20.600917 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.601557 25099 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.602994 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.603787 25102 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.604514 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.605212 25105 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.606724 24948 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:20.613466 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.614145 25103 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.617753 25028 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:20.618067 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.619865 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.620205 25098 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:20.626901 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:20.631812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.632479 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.636600 24852 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:20.630002 26568 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.643998 26438 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:20.668311 26438 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:20.653411 26562 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.672286 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.657898 26481 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.668167 26589 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240426 13:19:20.670465 26568 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:20.688359 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63045.5us
I20240426 13:19:20.688796 25028 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:20.691591 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:20.677569 26562 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.701352 26438 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.703505 26553 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.704890 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:20.705473 24948 raft_consensus.cc:595] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.706043 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.687510 26568 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.697261 26467 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:20.683840 26413 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.698657 26562 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:20.683743 26481 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.722229 26553 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.703958 26600 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240426 13:19:20.725642 26561 raft_consensus.cc:979] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:20.726117 26561 raft_consensus.cc:1067] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:20.728461 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.729135 25096 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.730965 26481 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.731359 26467 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.732174 26467 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.738078 24953 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:20.742219 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.743562 24950 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.748348 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.749317 26291 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:20.749639 26291 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.749840 26291 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:20.750890 24950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:20.751659 24950 raft_consensus.cc:595] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.753605 25028 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.754966 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:20.760162 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81360.5us
I20240426 13:19:20.763471 24954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:20.761818 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:20.764047 24954 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:20.772470 24954 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.782478 26441 raft_consensus.cc:979] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.764372 26526 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:20.810659 24954 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.821655 26610 raft_consensus.cc:979] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.822202 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:20.823017 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:20.823732 26527 raft_consensus.cc:979] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.801496 26291 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:20.855813 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62532.6us
I20240426 13:19:20.855849 24954 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:20.857334 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61162us
I20240426 13:19:20.873209 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.873888 24954 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:20.794445 26611 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240426 13:19:20.874696 26551 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.875144 26551 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.877025 26551 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.878490 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.879051 25096 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.878886 26562 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.879438 25096 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:20.875411 26610 raft_consensus.cc:1067] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.803642 26553 raft_consensus.cc:1067] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.890733 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.892442 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.893162 25023 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.896670 26291 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.907557 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.927561 24948 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.906229 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.939721 24954 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:20.930963 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:20.945514 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.922852 26563 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:19:20.946456 25025 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:20.961776 26291 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.902689 25096 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.009497 26588 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:21.009577 25028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:21.010242 25028 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.010584 25028 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.010656 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:20.871107 26526 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.906116 26431 raft_consensus.cc:1067] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.972481 26413 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.994765 26568 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.003679 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.012199 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:21.009974 26588 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:21.026082 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:21.109267 24950 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.109714 25025 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.116870 25025 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.109948 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.122021 25100 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:21.124752 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98743.6us
I20240426 13:19:21.124472 26291 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.126257 25103 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.135130 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132647us
I20240426 13:19:21.136561 26291 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.171779 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:21.134343 26614 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.177799 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.269479 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:21.177830 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56003us
I20240426 13:19:21.121421 26481 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:21.171587 26473 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.252413 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.262192 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 235921us
I20240426 13:19:21.278321 25028 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.299324 25028 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.266173 25023 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.267562 25024 raft_consensus.cc:2354] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:21.129714 26562 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.275023 25104 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.131290 26551 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.280756 25025 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.310463 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:21.287264 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.010s	sys 0.017s
I20240426 13:19:21.331400 25025 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:21.340056 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.011s	sys 0.018s
I20240426 13:19:21.312291 26291 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:21.276543 26515 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.311045 26551 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:21.316504 26481 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.370373 26481 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.357065 26610 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.334295 26483 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:21.379326 26610 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.312084 26614 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.351083 25026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:21.385613 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.359709 26551 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:21.395072 25102 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:21.396513 25102 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:21.311683 26473 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.352660 26515 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.294449 26588 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.404615 26524 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.430833 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.426785 26413 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.435290 25104 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.413518 25026 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.413432 25102 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.338275 26526 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.410521 26562 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.442348 25026 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.443756 25102 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:21.444185 26619 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20240426 13:19:21.446105 26610 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.442404 26413 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.387104 26483 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.473594 26483 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.457187 26473 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.458369 25100 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.469143 26622 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.335s	user 0.000s	sys 0.010s
I20240426 13:19:21.478197 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.479830 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.470161 26614 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:21.471807 26620 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20240426 13:19:21.478964 25100 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.494563 25102 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.433410 26568 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.497628 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.505712 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.498468 26617 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.510511 25104 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.513273 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.473603 26413 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.454499 26524 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:21.526582 26524 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.523295 26450 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.529227 26450 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.530063 26450 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.531103 26588 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.493023 26562 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.532167 26618 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:21.532691 26618 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.533668 26562 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.514951 26617 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.533538 26618 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.534740 26617 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.510068 26431 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.510790 26483 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:21.559248 25026 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.548617 26562 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.551169 26431 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.568113 26431 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.551105 26483 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.588582 25096 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.560266 26515 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.591320 24854 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:21.606983 26483 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.637712 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.642781 25105 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:21.644582 25096 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.643123 25105 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:21.595922 26562 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.661469 25105 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.682857 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.661530 25026 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.702318 26291 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.702988 26291 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.711581 26291 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.712612 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:21.713706 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 426916us
I20240426 13:19:21.710287 26588 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.710906 26473 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.743834 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.751205 26602 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.777420 26625 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.777899 26625 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.775589 26602 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.756587 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.766952 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.770522 25617 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:21.774124 25026 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:21.781070 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.785636 25104 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.765816 26595 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.790616 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.792306 25102 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:21.779914 26625 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.805496 26467 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.806600 26467 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.765689 26588 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.770154 26473 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.807651 26602 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.803244 26625 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.821362 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.013s	sys 0.025s
W20240426 13:19:21.821749 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.013s	sys 0.025s
W20240426 13:19:21.831073 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.012s	sys 0.053s
W20240426 13:19:21.831612 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.012s	sys 0.053s
I20240426 13:19:21.843677 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:21.846285 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.004s	sys 0.029s
W20240426 13:19:21.847096 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.004s	sys 0.029s
I20240426 13:19:21.850423 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.819212 26562 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.852356 26467 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.862252 26291 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.862811 26291 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.860524 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:21.796705 26595 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.864269 25615 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.864480 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.867240 24948 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.869741 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.870865 24948 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.856153 26624 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.871155 26291 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.821909 26625 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.856338 26441 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.872696 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53337.6us
I20240426 13:19:21.874966 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.876498 24955 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:21.876765 24955 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:21.862717 26568 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.865136 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.024s	sys 0.027s
I20240426 13:19:21.878834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:21.880230 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.025s	sys 0.027s
I20240426 13:19:21.863410 26632 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.865325 26635 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.880548 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 570133us
I20240426 13:19:21.880465 25102 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.878746 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.881830 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.889899 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.890331 25028 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:21.890426 24951 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.882347 24956 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.890853 25025 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:21.882808 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.883562 26534 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20240426 13:19:21.893764 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.894362 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.892560 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.897629 24921 leader_election.cc:398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:21.897980 24921 leader_election.cc:401] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:21.898430 25623 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:21.899909 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.900476 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.900859 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:21.902774 25615 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.884745 24955 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.903604 24955 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.885561 26632 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:21.906122 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.898630 25616 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:21.883934 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.906708 25027 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:21.909415 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.909163 24948 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.910154 24956 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:21.911760 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.901947 26611 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.917678 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.914295 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:21.879554 26441 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.876125 26624 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.917229 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:21.918561 24948 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.886077 26568 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.920743 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.943411 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.945241 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.908888 26632 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:21.954916 25025 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:21.947093 25027 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:21.947741 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.998157 25030 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:21.947903 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.912012 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.948402 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.948513 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:21.948637 26441 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.949240 25024 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.914563 26588 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.919981 26618 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:22.015935 26630 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.993096 24955 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.993562 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74455.6us
I20240426 13:19:21.986968 26291 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.041975 26630 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:21.915349 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.995466 26641 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20240426 13:19:21.995362 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.919881 26473 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.986562 24950 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.053807 25025 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:22.058293 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.058424 26595 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.059104 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.059726 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.001215 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:22.042497 26291 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.914702 26640 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.902853 26635 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.047618 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.047899 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.053084 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:22.111173 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:22.111558 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:22.107966 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 60430.1us
I20240426 13:19:22.112988 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:22.107244 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65027.5us
I20240426 13:19:21.997421 26617 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.000849 26553 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.922156 26611 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.052757 26505 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.192719 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:19:22.198159 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197061us
I20240426 13:19:22.053913 25616 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:22.198540 26291 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.209498 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.210620 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.212654 25025 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.111059 26640 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.212512 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:22.220518 24954 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:22.225617 26291 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:22.225888 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:22.191329 26467 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.208544 26553 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.229691 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.226725 25096 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:22.192354 26441 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240426 13:19:22.223107 26640 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.232333 26467 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.285637 26467 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.226887 26526 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.286276 26526 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.288112 26526 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.239290 26553 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.292196 26467 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.294142 26467 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.263733 26618 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:22.224258 26505 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.232583 26625 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:22.259668 26646 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.305s	user 0.000s	sys 0.000s
I20240426 13:19:22.295524 26553 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.296945 26551 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.871s	user 0.011s	sys 0.000s
W20240426 13:19:22.320219 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 274574us
W20240426 13:19:22.329571 26649 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.277s	user 0.001s	sys 0.000s
W20240426 13:19:22.329705 26642 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
I20240426 13:19:22.330684 26630 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:22.331136 26624 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.333068 24951 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:22.351402 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 137551us
I20240426 13:19:22.341287 25024 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240426 13:19:22.345834 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 155765us
W20240426 13:19:22.360255 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 132747us
I20240426 13:19:22.328362 26508 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.363761 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:22.383422 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:22.387830 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:22.392385 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.393311 24948 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:22.396812 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.398257 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.362130 26568 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.404992 26626 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:22.405198 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:22.405495 26654 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.406245 26635 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.407408 24952 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:22.413771 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.347577 26551 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.422009 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:22.414620 24955 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.400799 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.392565 26508 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.423266 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 197446us
I20240426 13:19:22.451982 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.485314 25029 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:22.486516 26508 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:22.488099 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 222117us
I20240426 13:19:22.489045 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:22.437884 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.492153 25617 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:22.493580 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:22.499632 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.500728 25103 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:22.501253 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.501868 25096 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:22.504727 25096 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:22.513659 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88292.7us
I20240426 13:19:22.423285 26654 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.554931 26654 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.296082 26618 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.583714 26669 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:22.586655 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 93144.2us
W20240426 13:19:22.437129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 137296us
W20240426 13:19:22.598584 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 234919us
I20240426 13:19:22.576277 26481 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.415019 26586 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:22.628561 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 138179us
I20240426 13:19:22.594161 26625 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.591213 26611 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.627701 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:22.584215 26669 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:22.593000 26508 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.652763 26611 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.655743 26663 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:22.656139 26526 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.630944 26586 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.656388 26656 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:22.656780 26648 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.392s	user 0.000s	sys 0.001s
I20240426 13:19:22.657066 26601 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.657377 26659 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:22.657761 26623 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20240426 13:19:22.656201 26660 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.653829 26669 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:22.630250 26553 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.658154 26505 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.650532 26635 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.631770 26481 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.690760 26481 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.667963 26678 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.691540 26678 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.691953 26678 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.684515 26659 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.694850 26659 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.669656 25105 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:22.670192 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 71797.5us
I20240426 13:19:22.671064 26665 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.663772 25616 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.700502 26665 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.702661 26665 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.631986 26625 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.703266 26625 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.705088 26617 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:22.678826 26645 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.762s	user 0.005s	sys 0.003s
I20240426 13:19:22.705607 26617 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.679226 26551 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.677460 26586 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.683553 26601 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.707433 26601 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.678102 26526 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.678324 26483 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.036s	user 0.011s	sys 0.000s
I20240426 13:19:22.680316 26656 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.658435 26611 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.661470 26664 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.687021 26660 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.688336 26635 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.722769 26586 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.668548 26508 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.877642 25105 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.881110 26661 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.881335 26562 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.881865 26661 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.882012 26562 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.881188 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:22.729620 26656 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.896224 26551 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.896728 26551 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.897111 26551 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.904525 26562 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.913585 26508 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.887440 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1255) took 1261 ms. Trace:
I20240426 13:19:22.915241 25105 rpcz_store.cc:276] 0426 13:19:21.619294 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.619429 (+   135us) service_pool.cc:225] Handling call
0426 13:19:22.654775 (+1035346us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b58003101b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:19:22.880681 (+225906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139556224}
I20240426 13:19:22.729933 26483 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.152951 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.008s	sys 0.002s
W20240426 13:19:23.485831 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.010s	sys 0.007s
W20240426 13:19:23.012367 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 423422us
W20240426 13:19:23.183581 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.005s	sys 0.030s
I20240426 13:19:22.925295 25622 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:23.152743 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.026s	sys 0.021s
W20240426 13:19:23.495303 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.026s	sys 0.029s
W20240426 13:19:23.487675 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.008s	sys 0.041s
I20240426 13:19:22.856907 26611 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:23.270577 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.008s	sys 0.013s
W20240426 13:19:23.498123 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.016s	sys 0.033s
W20240426 13:19:23.402556 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.023s	sys 0.017s
W20240426 13:19:23.499322 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.023s	sys 0.019s
W20240426 13:19:23.493815 24948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.006s	sys 0.012s
W20240426 13:19:23.500785 24948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.006s	sys 0.012s
W20240426 13:19:22.918969 26632 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.926s	user 0.009s	sys 0.000s
I20240426 13:19:23.500604 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.501605 25027 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:23.242197 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.013s	sys 0.063s
W20240426 13:19:23.239195 26640 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.954s	user 0.007s	sys 0.004s
I20240426 13:19:23.504364 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 449) took 1638 ms. Trace:
W20240426 13:19:23.481361 26656 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.000s
W20240426 13:19:23.423310 26617 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.717s	user 0.009s	sys 0.003s
I20240426 13:19:23.504725 25623 rpcz_store.cc:276] 0426 13:19:21.859768 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.859902 (+   134us) service_pool.cc:225] Handling call
0426 13:19:21.860162 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:21.898782 (+ 38620us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:23.494606 (+1595824us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:23.497799 (+  3193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83563392}
W20240426 13:19:23.141587 25026 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.796s	user 0.015s	sys 0.027s
W20240426 13:19:23.489593 26618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.013s	sys 0.005s
W20240426 13:19:23.497550 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.006s	sys 0.005s
I20240426 13:19:23.509886 25022 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
W20240426 13:19:23.428153 26654 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.873s	user 0.012s	sys 0.000s
I20240426 13:19:22.922345 25624 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:23.204550 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.069s	sys 0.133s
I20240426 13:19:23.512284 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:23.564177 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.049s	user 0.069s	sys 0.135s
I20240426 13:19:23.512593 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:23.514226 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.115s	user 0.021s	sys 0.064s
W20240426 13:19:23.564512 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.04993e+06us
I20240426 13:19:23.564756 25025 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.886185 26664 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.920466 26678 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.287484 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.917755 26586 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.916088 26635 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.515636 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.922070 26660 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.557361 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.484979 26562 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:23.556344 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.006s	sys 0.005s
W20240426 13:19:23.560024 26618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.013s	sys 0.006s
I20240426 13:19:23.371758 26665 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:23.384109 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.648s (SENT) [suppressed 2 similar messages]
W20240426 13:19:23.557101 25026 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.180s	user 0.019s	sys 0.039s
I20240426 13:19:23.569469 25027 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:23.173694 26601 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.565524 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.570030 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:23.204200 26676 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.000s	sys 0.010s
I20240426 13:19:23.099735 26661 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.965708 26625 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.922991 26669 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.584832 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 941505us
W20240426 13:19:23.490295 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.008s	sys 0.003s
W20240426 13:19:23.617424 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.009s	sys 0.003s
I20240426 13:19:23.504515 26640 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.592353 25623 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:23.592000 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:23.596812 26683 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.600600 26658 raft_consensus.cc:979] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:23.602238 26689 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:23.602494 26601 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.561921 26654 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.604887 26483 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.573523 26677 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:23.609575 26678 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.508884 26632 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.592000 26660 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.581231 25024 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:23.618619 26551 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.721s	user 0.007s	sys 0.005s
I20240426 13:19:23.591774 26562 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.538924 26656 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.625537 26697 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.631927 26689 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.612097 26665 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.642181 26692 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:19:23.629781 26683 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.646322 26654 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.617298 26618 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.648335 26684 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:23.620239 26688 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.589927 26635 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:23.747450 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 169) took 1897 ms (client timeout 1620 ms). Trace:
I20240426 13:19:23.675455 24956 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:23.834622 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1242) took 1440 ms. Trace:
W20240426 13:19:23.831154 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.100s	sys 0.104s
W20240426 13:19:23.706928 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 454) took 1335 ms (client timeout 1645 ms). Trace:
W20240426 13:19:23.895277 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.100s	sys 0.113s
W20240426 13:19:23.714907 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 330879us
W20240426 13:19:23.895562 25622 rpcz_store.cc:269] 0426 13:19:22.368139 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.368262 (+   123us) service_pool.cc:225] Handling call
0426 13:19:23.703173 (+1334911us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b540033a254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.703344 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1179458432}
I20240426 13:19:23.545676 26617 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.745206 25023 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240426 13:19:23.707597 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1182) took 1782 ms (client timeout 1869 ms). Trace:
I20240426 13:19:23.897668 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:19:23.896157 24950 rpcz_store.cc:269] 0426 13:19:21.850118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.850229 (+   111us) service_pool.cc:225] Handling call
0426 13:19:23.747238 (+1897009us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.747359 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305292544}
I20240426 13:19:23.647753 26601 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:23.595261 26676 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.002s	sys 0.011s
W20240426 13:19:23.912811 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 293206us
W20240426 13:19:23.886350 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 388050us
I20240426 13:19:23.803385 26467 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.928330 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:23.868336 26507 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.748642 26664 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.852850 26589 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:23.911057 25031 rpcz_store.cc:269] 0426 13:19:21.921830 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.921947 (+   117us) service_pool.cc:225] Handling call
0426 13:19:23.704454 (+1782507us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.704572 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258830080}
I20240426 13:19:23.957144 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:23.980180 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:23.980984 25023 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:23.983716 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:23.742362 26625 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.962347 26683 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.744091 26699 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.907094 26669 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.867465 26354 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.898980 26551 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.899385 26660 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.899521 26688 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.654042 26677 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.906497 26697 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:24.325891 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 254170us
I20240426 13:19:24.328414 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:23.984143 25104 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:24.328783 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.335090 25031 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.335594 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:24.095264 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:24.118116 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.014s	sys 0.059s
I20240426 13:19:23.913795 26535 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.627156 26661 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:24.117928 25030 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.032s	sys 0.027s
W20240426 13:19:24.337074 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.023s	sys 0.067s
I20240426 13:19:24.341812 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:24.287791 26694 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.573s	user 0.001s	sys 0.001s
W20240426 13:19:24.345533 25030 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.045s	sys 0.038s
I20240426 13:19:24.058993 25618 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:24.256058 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:23.955152 26685 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:24.317341 26611 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.760s	user 0.007s	sys 0.003s
I20240426 13:19:23.958227 26640 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.352892 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.197026 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.351652 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.351972 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.354254 25023 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:24.354611 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.196862 26534 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.039587 26633 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.135798 26588 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.287122 26689 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.101552 26634 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.101840 26527 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.327636 26526 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.083197 26473 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.060195 26515 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.350016 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.286170 26656 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.088786 26604 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.973707 26635 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.951225 26601 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.323817 26665 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.071262 26614 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.355357 25024 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:24.353698 25100 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.090641 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:23.916347 24948 rpcz_store.cc:276] 0426 13:19:22.394085 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.394218 (+   133us) service_pool.cc:225] Handling call
0426 13:19:22.394508 (+   290us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:22.406897 (+ 12389us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:22.406927 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:22.681990 (+275063us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:19:22.682421 (+   431us) log.cc:842] Serialized 26 byte log entry
0426 13:19:23.501299 (+818878us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:23.501349 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:23.501463 (+   114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:23.834246 (+332783us) raft_consensus.cc:1671] finished
0426 13:19:23.834273 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:23.834526 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122120,"raft.queue_time_us":30533,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":10601,"spinlock_wait_cycles":4230528,"thread_start_us":817888,"threads_started":1,"wal-append.queue_time_us":1129657}
W20240426 13:19:24.432207 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.039s	sys 0.059s
W20240426 13:19:24.434579 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.040s	sys 0.060s
I20240426 13:19:24.085641 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.347558 26685 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.339491 26562 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.955844 26618 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.965369 26467 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.356323 26611 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.984647 26710 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.388635 26679 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.408263 26508 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.364424 26633 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.352200 26617 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.059049 26589 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.386410 26625 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:24.393482 26687 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.517s	user 0.002s	sys 0.000s
I20240426 13:19:24.395313 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.477603 25102 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:24.449142 26654 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:24.396382 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.085809 26672 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.196337 26711 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.328943 26682 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.123303 26632 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.987982 26507 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:24.359589 26658 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.004s	sys 0.039s
W20240426 13:19:23.987685 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 491290us
I20240426 13:19:24.437415 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.444931 26604 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.544243 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.544869 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:24.545274 26604 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.550994 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 460) took 1042 ms. Trace:
I20240426 13:19:24.551548 25615 rpcz_store.cc:276] 0426 13:19:23.508535 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.508862 (+   327us) service_pool.cc:225] Handling call
0426 13:19:23.509212 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:23.956835 (+447623us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:23.956878 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:23.956933 (+    55us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:23.956975 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:24.550708 (+593733us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:24.550769 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:24.550926 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1028965120}
I20240426 13:19:24.552804 24950 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:24.418061 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.435581 25103 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:23.964928 26704 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 13:19:24.384459 26640 raft_consensus.cc:1067] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:24.439939 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.440994 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.438679 26413 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.442845 26697 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.381404 26588 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.438972 26441 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.443073 26688 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.349289 26535 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.355381 26661 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.385696 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.433251 26673 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.433799 26705 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.409426 26683 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.437531 26689 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.462916 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:24.466387 26714 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.466539 26713 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.439256 26526 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.437949 26503 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.410122 26660 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.436905 26656 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.411970 26626 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.474823 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.443086 26551 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.444756 26635 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.338732 26354 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.486112 25100 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:24.362856 26534 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.492308 26666 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.492432 26712 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.521917 25031 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:24.409525 26527 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.411672 26515 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.398365 26634 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.460707 26676 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.439801 26614 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.348496 26699 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.411501 26473 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.476483 26508 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.526566 26625 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.487260 26679 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.461380 26710 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.868213 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1188) took 1355 ms. Trace:
W20240426 13:19:24.876770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 413970us
I20240426 13:19:24.882099 24954 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:24.553686 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:24.884792 25104 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:24.886860 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.887712 25025 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:24.890779 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:24.543359 26658 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.004s	sys 0.044s
W20240426 13:19:24.892375 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 349445us
W20240426 13:19:24.891259 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1236) took 2840 ms (client timeout 1504 ms). Trace:
I20240426 13:19:24.876881 25030 rpcz_store.cc:276] 0426 13:19:23.512540 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.514269 (+  1729us) service_pool.cc:225] Handling call
0426 13:19:23.562229 (+ 47960us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:24.346375 (+784146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.346414 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:24.354178 (+  7764us) log.cc:842] Serialized 26 byte log entry
0426 13:19:24.446839 (+ 92661us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:24.446871 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:24.446933 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:24.867905 (+420972us) raft_consensus.cc:1671] finished
0426 13:19:24.867931 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:24.868124 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17652,"raft.queue_time_us":186171,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"thread_start_us":847151,"threads_started":2,"wal-append.queue_time_us":511903}
W20240426 13:19:24.895669 24956 rpcz_store.cc:269] 0426 13:19:21.994173 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.994320 (+   147us) service_pool.cc:225] Handling call
0426 13:19:24.834596 (+2840276us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54001018d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:24.834739 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3535935744}
I20240426 13:19:24.909072 25099 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.872478 26467 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.865592 26601 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.762647 26562 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:24.706670 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 150) took 3011 ms (client timeout 1648 ms). Trace:
I20240426 13:19:24.912228 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:24.975595 25622 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:24.801270 26595 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.870448 26618 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.797076 26685 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.887202 26617 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:24.968780 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.968408 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.888641 26551 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.874161 26677 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.537111 26654 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:24.838227 26681 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.891894 26635 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.761528 26709 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.761137 26722 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.980124 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:24.889067 26626 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.981627 25025 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.012373 26709 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.014020 26626 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.014940 26681 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.909670 26666 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.017616 26681 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.837881 26718 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.913604 25096 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:24.972681 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.018132 26666 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.557507 26682 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.977821 26589 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.018237 26718 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.019212 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 200201us
I20240426 13:19:25.020450 26682 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:24.979007 25022 rpcz_store.cc:269] 0426 13:19:21.682315 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.682533 (+   218us) service_pool.cc:225] Handling call
0426 13:19:24.693424 (+3010891us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:24.693586 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4199719296}
I20240426 13:19:25.020495 26718 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.021946 26660 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.898157 26673 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:25.022507 26660 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.024372 26673 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.932559 26658 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.012398 26722 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.025458 26658 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.027002 26722 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.907991 26664 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.027695 26664 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.028168 26664 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.028343 26658 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.871609 26507 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.021926 25025 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:24.996688 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 534164us
I20240426 13:19:24.541548 26672 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.024516 26660 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.962985 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.968315 26719 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.894361 26688 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20240426 13:19:24.883777 26689 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.037987 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:24.890470 26441 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.887315 26526 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.043010 26710 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.889969 26413 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.980021 26640 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.063678 26291 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:24.891816 26683 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.895608 26656 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.046831 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.004s	sys 0.028s
W20240426 13:19:25.067787 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.004s	sys 0.028s
I20240426 13:19:25.031178 26601 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.029836 26611 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.064862 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.029735 26633 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.894270 26503 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.913048 26676 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.069669 26534 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.070864 26634 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.962272 26713 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.983342 26595 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.071890 26588 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.075726 26720 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.076277 26721 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.076702 26526 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.076961 26708 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.970667 26714 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.117942 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.009083 26551 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.046113 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 453) took 2669 ms (client timeout 2235 ms). Trace:
I20240426 13:19:25.083531 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.889122 26705 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.032729 26635 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.641642 26711 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.088277 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 172) took 3087 ms. Trace:
I20240426 13:19:24.911105 26712 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.090749 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.091787 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.092361 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.095249 26672 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.096153 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.140341 24950 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:25.072525 26660 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:24.988377 26562 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.099963 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.075177 26589 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:25.115542 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.093353 26710 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20240426 13:19:25.072310 26640 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.079514 26601 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.122442 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.014s	sys 0.036s
I20240426 13:19:25.123310 26413 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:25.134555 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 219108us
I20240426 13:19:25.137634 26595 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.143369 25023 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.147346 24954 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.130426 24852 rpcz_store.cc:276] 0426 13:19:21.994985 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.995772 (+   787us) service_pool.cc:225] Handling call
0426 13:19:25.082616 (+3086844us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240426 13:19:25.163502 25031 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:25.106659 26503 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.149641 26354 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.139941 26515 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.120118 26527 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.116954 26721 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.099577 26654 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:25.154505 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.014s	sys 0.036s
I20240426 13:19:25.126652 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.122985 26535 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.195667 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.050360 26719 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.196349 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.208712 25622 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:25.198796 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.210743 25028 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.212767 25027 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:25.198386 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.216044 25022 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:25.217430 26716 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:25.122429 25624 rpcz_store.cc:269] 0426 13:19:22.367720 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.389191 (+ 21471us) service_pool.cc:225] Handling call
0426 13:19:25.030299 (+2641108us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.036781 (+  6482us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989924096}
I20240426 13:19:25.154925 24949 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.121078 26709 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.172528 26705 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.165565 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.161785 26656 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.166803 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.226174 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 274) took 1143 ms. Trace:
I20240426 13:19:25.169545 26712 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:25.127432 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1647238229999175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20240426 13:19:25.171196 26635 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.154070 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.173031 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.244092 25100 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:25.221602 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.127264 26661 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.162012 26508 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.226708 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.225915 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.167997 26473 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.144205 26660 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.229012 24954 raft_consensus.cc:2354] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.150915 26711 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.206733 26614 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.217952 26676 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.237502 26683 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.229704 26689 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.154145 26708 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.164096 26679 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.148717 26625 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.276691 26719 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.278077 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.245505 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:25.246434 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.225188 26617 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.244158 25103 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:25.246831 26713 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.248556 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 213) took 1139 ms. Trace:
I20240426 13:19:25.249946 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 210) took 1165 ms. Trace:
I20240426 13:19:25.206085 26666 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.253317 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.253660 26721 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.254654 26640 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.229987 26441 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:25.238816 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 268) took 3353 ms (client timeout 2168 ms). Trace:
I20240426 13:19:25.256983 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.294749 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.221410 26716 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.231062 26699 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.261015 26562 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.262027 26601 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.264443 26551 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.252543 26660 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:25.265313 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 214) took 1158 ms. Trace:
I20240426 13:19:25.265666 26664 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.311187 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.268196 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.268317 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.312341 25025 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.267915 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.313097 25102 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:25.268831 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.269434 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.270334 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.314093 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.314472 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.314800 25102 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:25.315825 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.309844 25097 rpcz_store.cc:276] 0426 13:19:24.106922 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.281393 (+174471us) service_pool.cc:225] Handling call
0426 13:19:24.344945 (+ 63552us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.913342 (+568397us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.913378 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.913430 (+    52us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:24.913479 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.264941 (+351462us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b540009ced4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.265000 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.265141 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306922240}
I20240426 13:19:25.271173 25099 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.141517 26720 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.272330 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.319785 24954 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.317178 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:25.320559 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.273281 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.272334 26689 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.274876 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:25.280452 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:25.324620 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.280071 25098 rpcz_store.cc:276] 0426 13:19:24.084609 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.085722 (+  1113us) service_pool.cc:225] Handling call
0426 13:19:24.834081 (+748359us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:24.834275 (+   194us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:25.110781 (+276506us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.110813 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:25.110856 (+    43us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:25.110885 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.249653 (+138768us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54001b3854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.249711 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.249854 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":648963072}
I20240426 13:19:25.282955 25029 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.283236 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:25.283814 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 193165us
I20240426 13:19:25.326056 25622 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:25.300312 26690 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.279945 25101 rpcz_store.cc:276] 0426 13:19:24.085483 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.277390 (+191907us) service_pool.cc:225] Handling call
0426 13:19:24.351784 (+ 74394us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.704458 (+352674us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.704513 (+    55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.704568 (+    55us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:24.704611 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.222089 (+517478us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.222130 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.225142 (+  3012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810539392}
I20240426 13:19:25.278098 26714 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.295884 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.306949 25023 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.308264 24920 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:25.309713 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 422847us
I20240426 13:19:25.231921 25616 rpcz_store.cc:276] 0426 13:19:24.083264 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.085249 (+  1985us) service_pool.cc:225] Handling call
0426 13:19:24.286442 (+201193us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.356537 (+ 70095us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.356578 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.360221 (+  3643us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:24.383311 (+ 23090us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.225864 (+842553us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.225965 (+   101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.226107 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152181120}
I20240426 13:19:25.339712 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.273808 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:25.340240 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.322396 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.341262 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:25.322547 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.324822 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.284050 24956 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.342598 25028 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.340281 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.326912 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.328262 25615 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240426 13:19:25.326967 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.344476 24953 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.344702 24948 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.285238 26635 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.329694 25103 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.337463 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:25.330309 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.309537 26625 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.351953 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.304589 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:25.340637 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66892.3us
I20240426 13:19:25.296902 26614 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:25.350571 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.336292 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.351395 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.354038 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.284917 26617 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.322782 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.358863 24956 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:25.361740 24995 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11fad756437243d2959f5998d6bf8309. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:19:25.437923 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 76365.4us
I20240426 13:19:25.297068 26666 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.428362 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1182) took 1108 ms. Trace:
I20240426 13:19:25.428537 25624 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.428992 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.471889 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.468659 25099 rpcz_store.cc:276] 0426 13:19:24.320337 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.387562 (+ 67225us) service_pool.cc:225] Handling call
0426 13:19:25.428178 (+1040616us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.428303 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72873856}
I20240426 13:19:25.429874 26551 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:25.430537 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 122357us
I20240426 13:19:25.476320 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.476619 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.430835 26707 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.429880 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.370882 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.431730 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:25.295570 25618 rpcz_store.cc:269] 0426 13:19:21.885828 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.886511 (+   683us) service_pool.cc:225] Handling call
0426 13:19:25.238623 (+3352112us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.238755 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4274581760}
I20240426 13:19:25.433087 24947 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.433635 24954 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.368579 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.360714 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.434353 26716 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.361937 26714 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.439538 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.441406 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 458) took 2172 ms. Trace:
I20240426 13:19:25.440016 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.331403 26690 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.491439 26690 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.489332 26714 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.454592 24996 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:25.441371 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.492758 24996 leader_election.cc:401] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20240426 13:19:25.492998 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51698.1us
I20240426 13:19:25.493355 25103 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:25.493621 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.443641 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.429448 26656 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.496378 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:25.496826 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.010s	sys 0.031s
W20240426 13:19:25.497120 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.010s	sys 0.031s
I20240426 13:19:25.334153 26600 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:25.448922 25105 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.442967 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.326822 26588 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.480221 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.499251 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.501145 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.484012 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.499075 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.502236 25023 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.502992 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.481256 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.503180 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.485262 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.485706 25098 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.503983 24954 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.482529 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:25.486124 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117629us
I20240426 13:19:25.505484 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.488428 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.506105 24954 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.507809 25622 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:25.508582 25100 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.509408 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.510290 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.491874 26717 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.512378 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:25.497794 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1543) took 1049 ms. Trace:
I20240426 13:19:25.489996 25022 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.513039 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.519973 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.520980 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.521148 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.481858 26707 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:25.512509 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.524932 24948 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.513365 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.368672 26625 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.440156 26614 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.478065 25026 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.478282 26704 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20240426 13:19:25.511648 26588 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.491606 26553 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240426 13:19:25.511749 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.514380 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.548118 25031 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:25.548431 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.517901 24951 rpcz_store.cc:276] 0426 13:19:24.448686 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.448806 (+   120us) service_pool.cc:225] Handling call
0426 13:19:24.451367 (+  2561us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.451534 (+   167us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:24.736306 (+284772us) log.cc:842] Serialized 14 byte log entry
0426 13:19:25.497495 (+761189us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:25.497544 (+    49us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:25.497582 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.497645 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.497748 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":517455,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":3017,"raft.queue_time_us":759,"thread_start_us":1029786,"threads_started":2,"wal-append.queue_time_us":770733}
I20240426 13:19:25.356580 26737 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:25.510875 25100 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:25.525971 24956 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.526391 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:25.353276 26473 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.555106 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.522541 26700 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.532778 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.535655 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.535806 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.536628 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.529433 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.498224 26413 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:25.441632 26708 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.569954 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.570750 25102 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.541054 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:25.488612 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.542443 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.352154 26689 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.544275 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.544400 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.356436 26635 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.545090 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 270296us
I20240426 13:19:25.545405 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.592861 24947 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.545476 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.522500 26717 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.483184 26551 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.522241 26683 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.532533 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.537165 26441 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:25.532214 26707 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.531956 26508 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.538727 26614 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:25.629607 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.564536 24949 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.569294 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.561362 26473 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.583760 26738 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:25.583892 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.543849 26505 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.583667 26601 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.587280 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:25.646070 25101 raft_consensus.cc:595] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.573691 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:25.489543 25617 rpcz_store.cc:276] 0426 13:19:23.269445 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.270127 (+   682us) service_pool.cc:225] Handling call
0426 13:19:24.389879 (+1119752us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:24.390098 (+   219us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.968373 (+578275us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.968408 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.968455 (+    47us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:24.968496 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.441087 (+472591us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.441153 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.441316 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1366731776}
I20240426 13:19:25.592785 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:25.511715 26656 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.593530 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:25.651252 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58550.4us
I20240426 13:19:25.550011 26588 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.597174 25098 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:25.556283 26737 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.653776 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.599135 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.601936 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:25.605408 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.655249 25029 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:25.607512 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.609751 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.609196 24854 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 277) took 4083 ms. Trace:
I20240426 13:19:25.585103 26718 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.618590 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:25.615568 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.568508 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:25.564205 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.675606 26696 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.186s
I20240426 13:19:25.595837 26413 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:25.599716 26717 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.636572 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.597671 26720 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.640103 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.640619 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.634342 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:25.642614 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119171us
I20240426 13:19:25.586109 26716 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:25.568914 26700 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.604619 26664 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.591008 25102 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.647789 26601 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.648232 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.629352 26508 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.636448 26473 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.631994 26441 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:25.670775 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68957.2us
I20240426 13:19:25.674203 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.655304 25588 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.821982 25588 leader_election.cc:401] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:25.676110 26653 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.676743 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.823494 25029 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.677636 26635 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.717276 26551 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.720391 26625 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.720530 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.798710 26601 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.725040 26656 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.739370 24947 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:25.866741 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:25.741025 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 176980us
I20240426 13:19:25.868289 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.868013 26413 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.869467 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:25.630499 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97637.6us
I20240426 13:19:25.754185 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.784871 25070 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:25.872253 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 105214us
I20240426 13:19:25.872709 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:25.787739 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.753979 26657 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.874043 26698 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:25.874459 26657 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:25.874846 26657 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.874133 26481 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.875484 26562 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.875538 26664 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:25.874164 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86927.3us
I20240426 13:19:25.877048 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.678617 26553 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.666231 26737 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:25.665401 26617 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.665699 26600 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.823940 25025 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:25.824692 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.684006 26666 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.783418 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:25.826614 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 171445us
I20240426 13:19:25.727514 26714 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.888988 26664 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:25.820649 25616 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:25.721660 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 375055us
I20240426 13:19:25.834314 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.834764 24956 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.814016 26508 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.959607 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.835453 26744 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.846738 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.865108 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.799108 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.880085 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:25.983440 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103449us
I20240426 13:19:25.888269 26553 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:25.948017 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68414.6us
I20240426 13:19:25.955200 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.917594 26600 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.988883 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.833930 26653 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.995465 26441 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.989395 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.743609 26718 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.989948 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.993208 26600 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.994874 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.996259 26653 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.990125 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.981581 26562 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:25.997341 26553 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.998026 26720 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.016017 26610 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.035775 26698 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.041239 26698 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.008735 26718 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:26.047358 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 129420us
W20240426 13:19:26.048063 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59270.3us
W20240426 13:19:26.047655 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 170693us
I20240426 13:19:26.051060 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.052210 26689 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.053171 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.072082 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.072674 24954 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:26.073449 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.074013 24954 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.055732 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.057404 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.060026 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.061496 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.063751 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.009374 26562 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.068768 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:26.070191 25588 leader_election.cc:398] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.074568 24956 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:26.075270 24947 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:26.077227 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.996668 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.080380 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.080821 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:26.068754 24949 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:26.081753 25588 leader_election.cc:401] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.084331 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.044898 26720 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.105636 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.108904 26708 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.047863 26716 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:26.114187 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44165.2us
I20240426 13:19:26.114939 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.117715 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.083482 26633 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.109884 26689 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:25.656771 24854 rpcz_store.cc:276] 0426 13:19:21.525781 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.528160 (+  2379us) service_pool.cc:225] Handling call
0426 13:19:25.609141 (+4080981us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:21.896368 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:21.987080 (+ 90712us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:21.993371 (+  6291us) write_op.cc:427] Acquired schema lock
0426 13:19:21.999430 (+  6059us) tablet.cc:660] Decoding operations
0426 13:19:22.059248 (+ 59818us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:22.059348 (+   100us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:22.059389 (+    41us) tablet.cc:683] Acquiring locks for 1 operations
0426 13:19:22.059605 (+   216us) tablet.cc:699] Row locks acquired
0426 13:19:22.059631 (+    26us) write_op.cc:260] PREPARE: finished
0426 13:19:22.725363 (+665732us) spinlock_profiling.cc:243] Waited 902 us on lock 0x7b2c000471f8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed08b6a 00007fda8ed087f3 00007fda8ed08791 00007fda90352e02 00007fda90352bdd 00007fda93d27d10 00007fda93d27828 00007fda93d2c111 00007fda93d2c0c9 00007fda93d2c059
0426 13:19:22.725431 (+    68us) write_op.cc:270] Start()
0426 13:19:22.725628 (+   197us) write_op.cc:275] Timestamp: P: 1714137562060674 usec, L: 0
0426 13:19:22.725651 (+    23us) op_driver.cc:331] REPLICATION: starting
0426 13:19:22.726244 (+   593us) log.cc:842] Serialized 535 byte log entry
0426 13:19:25.148568 (+2422324us) op_driver.cc:444] REPLICATION: finished
0426 13:19:25.448610 (+300042us) write_op.cc:300] APPLY: starting
0426 13:19:25.448703 (+    93us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:25.448816 (+   113us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:25.448844 (+    28us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:25.449070 (+   226us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:25.449253 (+   183us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 13:19:25.449313 (+    60us) write_op.cc:311] APPLY: finished
0426 13:19:25.449545 (+   232us) log.cc:842] Serialized 23 byte log entry
0426 13:19:25.449724 (+   179us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:25.449814 (+    90us) write_op.cc:446] Released schema lock
0426 13:19:25.449858 (+    44us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:25.450176 (+   318us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:25.450210 (+    34us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1628,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":291485,"apply.run_cpu_time_us":1852,"apply.run_wall_time_us":1852,"num_ops":1,"prepare.queue_time_us":260872,"prepare.run_cpu_time_us":22973,"prepare.run_wall_time_us":1611330,"raft.queue_time_us":645897,"raft.run_cpu_time_us":4487,"raft.run_wall_time_us":213198,"replication_time_us":2415761,"spinlock_wait_cycles":2075392,"thread_start_us":1282759,"threads_started":4,"wal-append.queue_time_us":899830}]]}
I20240426 13:19:26.121541 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:26.111575 26720 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.101840 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.104585 26617 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.085600 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.107897 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.133241 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.083711 26714 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20240426 13:19:26.140875 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55370.5us
I20240426 13:19:26.126361 26682 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.166890 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.112834 26747 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.118563 26744 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.157236 26595 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:26.193634 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47582.2us
I20240426 13:19:26.117141 26716 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.125579 26633 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.144141 26481 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.194578 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.203883 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.142283 26562 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.140017 26668 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.195920 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.207227 25105 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:26.197822 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.140379 26698 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.140914 26700 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.205796 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.217279 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.193563 26595 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.197407 26682 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:26.206871 26660 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.876s	user 0.006s	sys 0.011s
I20240426 13:19:26.188663 26747 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.230824 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.201885 26633 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.237916 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.230067 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.239035 24948 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.239759 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.231405 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.232112 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.201799 26481 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.221827 26684 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.226130 26595 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.240386 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.244424 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.216804 26664 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.226763 26700 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.227494 26676 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.249724 25619 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
W20240426 13:19:26.249852 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 238926us
I20240426 13:19:26.227716 26610 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:26.255460 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 384964us
I20240426 13:19:26.251667 25589 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.250753 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.237116 26747 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.258806 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.256055 26676 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.249019 26481 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.250171 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.228421 26698 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 13:19:26.237097 26688 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:26.250958 26684 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.251114 26681 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.252646 26700 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:26.255251 25624 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.756s	user 0.010s	sys 0.012s
I20240426 13:19:26.271155 25624 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:26.263056 26676 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.234813 26660 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.256736 26505 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.284821 26505 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.283900 26553 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.273108 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:26.286006 26660 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.286100 26699 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.286617 26699 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.235428 26617 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 13:19:26.274245 26745 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.270020 26473 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.264191 26668 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.267438 26684 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:26.285094 26751 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20240426 13:19:26.266758 26657 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.292491 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:26.295758 26707 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.003s
I20240426 13:19:26.301252 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.325368 26707 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.326197 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:26.327606 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 194451us
W20240426 13:19:26.320823 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63397.9us
I20240426 13:19:26.328841 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.272532 26700 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.331779 26481 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.332196 26682 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.332620 26682 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.332976 26682 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.323931 24949 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:26.287420 26699 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.328372 25620 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:26.327693 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.326792 26668 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.339154 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.341925 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.342739 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:26.343739 26755 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20240426 13:19:26.345906 26740 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.699s	user 0.002s	sys 0.000s
I20240426 13:19:26.345770 26505 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.347374 26657 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.349939 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.339365 26757 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.352499 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1539) took 1915 ms. Trace:
I20240426 13:19:26.350920 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.358345 26757 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.359725 26551 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.360139 26551 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.360301 26707 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.360549 26551 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.362215 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.336864 26481 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.365482 26601 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.356259 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.368193 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.352994 24955 rpcz_store.cc:276] 0426 13:19:24.437350 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.437716 (+   366us) service_pool.cc:225] Handling call
0426 13:19:24.875839 (+438123us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.043896 (+168057us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.043936 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.049254 (+  5318us) log.cc:842] Serialized 26 byte log entry
0426 13:19:25.539822 (+490568us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:25.539873 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.540002 (+   129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.345398 (+805396us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:26.347442 (+  2044us) raft_consensus.cc:1671] finished
0426 13:19:26.347465 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:26.352435 (+  4970us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22513,"raft.queue_time_us":98438,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6265,"spinlock_wait_cycles":48407552,"thread_start_us":657489,"threads_started":2,"wal-append.queue_time_us":546170}
I20240426 13:19:26.383117 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.362138 26727 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.387156 26668 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:26.387694 26505 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:26.366947 26588 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.693s	user 0.015s	sys 0.018s
I20240426 13:19:26.349505 26660 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:26.336864 26716 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.334026 26756 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.324582 26633 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.405025 24955 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.376767 26745 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.384155 26657 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.428411 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:26.396816 26754 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20240426 13:19:26.385694 26664 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:26.405395 26614 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.760s	user 0.010s	sys 0.005s
I20240426 13:19:26.446681 26664 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.366655 26696 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240426 13:19:26.395443 26727 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:26.451069 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110957us
I20240426 13:19:26.404691 26660 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:26.444878 26690 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.951s	user 0.017s	sys 0.002s
I20240426 13:19:26.371266 26595 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.441452 26716 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.426412 26745 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.424396 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.459149 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.440971 26657 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.468658 26595 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.468461 26657 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.466420 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.446185 26441 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240426 13:19:26.404542 26588 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.399943 26508 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.455963 26505 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.451869 26756 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.454042 26664 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.475548 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.478655 26473 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.460928 26660 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:26.487875 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:26.489888 26737 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.573s	user 0.025s	sys 0.024s
I20240426 13:19:26.490653 25030 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.490834 26553 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.500528 26633 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.500605 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.474879 26716 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.477247 26635 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.507980 26635 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.470908 26745 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.508821 26635 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.498667 26656 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:26.499198 26653 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.519379 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:26.519980 25099 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.491180 26625 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.523401 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.462966 26690 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.506651 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1244) took 1230 ms. Trace:
I20240426 13:19:26.507485 26601 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.517652 25022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:26.457710 26614 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.524560 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:26.494083 26505 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.526605 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.533821 25023 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.529359 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.537531 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.536717 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.502993 26551 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.539670 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.540656 25096 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:26.526216 24953 rpcz_store.cc:276] 0426 13:19:25.280873 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.353214 (+ 72341us) service_pool.cc:225] Handling call
0426 13:19:25.467865 (+114651us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.721372 (+253507us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.721430 (+    58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.740751 (+ 19321us) log.cc:842] Serialized 26 byte log entry
0426 13:19:26.050364 (+309613us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:26.050632 (+   268us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:26.502919 (+452287us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:26.502968 (+    49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.503014 (+    46us) raft_consensus.cc:1671] finished
0426 13:19:26.503028 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:26.506582 (+  3554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582402304,"thread_start_us":262571,"threads_started":1,"wal-append.queue_time_us":381491}
I20240426 13:19:26.554261 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.540310 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.530179 26679 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.504616 26737 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.527984 26660 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:26.569403 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44901.1us
I20240426 13:19:26.574963 26772 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:26.588397 26600 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.562s	user 0.014s	sys 0.004s
I20240426 13:19:26.588865 26600 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.524211 26775 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:26.536247 26588 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.579924 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.582624 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.583364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.490686 26668 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.593704 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.594897 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.589777 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.595271 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.596961 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.598075 25101 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.572824 26679 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.575348 26653 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.613898 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.589762 26716 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.589829 26772 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.616189 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.616458 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.621971 25097 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.614692 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.607698 26551 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.618921 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.626824 26676 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.631297 25101 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:26.631947 26508 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.632294 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.618665 25624 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.619100 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:26.637051 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 175472us
I20240426 13:19:26.639570 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.609162 26679 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.619100 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.645592 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.648633 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:26.617131 25026 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.066s	user 0.009s	sys 0.034s
W20240426 13:19:26.621385 26718 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.007s
W20240426 13:19:26.617609 26717 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.824s	user 0.015s	sys 0.005s
I20240426 13:19:26.660624 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.662004 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.664429 26666 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:26.664938 26761 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20240426 13:19:26.664952 26666 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.665563 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.666592 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.667065 25104 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.667201 26786 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:26.620303 26614 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.668699 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.633167 26772 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.669166 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.676461 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.677006 24955 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.619433 26660 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.640369 26653 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.693655 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.671623 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.673274 25029 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.668197 26786 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.670256 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.696599 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.683707 25619 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240426 13:19:26.575438 26756 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.676136 26688 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 13:19:26.667176 26666 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.704762 26700 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.706646 26700 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.708559 25022 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:26.709124 25022 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.708388 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.707763 25097 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:26.701646 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.711289 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.712339 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.715626 24955 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:26.719830 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50740.5us
I20240426 13:19:26.720222 26714 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.720378 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.720578 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.598397 26775 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.721458 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.722791 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.738047 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.738613 25028 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.651412 25026 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:26.722748 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.723664 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.742372 26291 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.724900 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.725661 26777 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.744706 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.725864 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:26.726240 26780 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20240426 13:19:26.743985 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.728981 26505 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.729116 26747 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.643328 26773 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20240426 13:19:26.753159 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.735343 26676 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.721951 24995 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
W20240426 13:19:26.699193 26778 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20240426 13:19:26.725906 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.741751 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.747083 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.718274 26653 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.709635 26786 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:26.717864 26657 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.745965 26656 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.718170 26551 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.755977 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.752744 25104 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:26.669432 26718 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.758373 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.766206 26505 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.669615 26717 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.773272 25615 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 13:19:26.774349 25027 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.724604 26625 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:26.752595 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 169) took 1480 ms. Trace:
I20240426 13:19:26.777776 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.751267 26690 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.750967 26553 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.782495 25028 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.781622 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.770928 26721 raft_consensus.cc:979] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:26.828248 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.722014 26635 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20240426 13:19:26.829717 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.781720 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.762566 25619 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:26.833101 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.733974 26684 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:26.875468 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 92027.8us
I20240426 13:19:26.758782 26785 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:26.771054 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.768671 26737 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:26.843643 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65935us
W20240426 13:19:26.847827 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 584212us
I20240426 13:19:26.862435 25617 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.884606 26745 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.886231 25616 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:26.851459 26505 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.776934 26786 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.871146 26653 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.888636 26786 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.864856 26772 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.866739 26718 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.868288 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:26.859297 26720 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.714s	user 0.014s	sys 0.014s
I20240426 13:19:26.889446 26704 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.893491 26720 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.895371 25027 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240426 13:19:26.869076 26682 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.536s	user 0.015s	sys 0.004s
I20240426 13:19:26.827827 26789 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:26.896282 26789 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.897706 25098 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:26.898278 26789 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.898763 26682 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.899259 26772 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.900597 26682 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.867738 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.903328 25101 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.873755 24952 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.842769 25026 rpcz_store.cc:276] 0426 13:19:25.266123 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.285310 (+ 19187us) service_pool.cc:225] Handling call
0426 13:19:26.746220 (+1460910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:26.872345 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.876186 24995 leader_election.cc:398] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.909276 24995 leader_election.cc:401] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.909447 25028 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:26.876129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94507.7us
I20240426 13:19:26.878721 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.878909 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.879587 26774 raft_consensus.cc:1067] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:26.912197 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 440) took 1085 ms. Trace:
I20240426 13:19:26.912580 25620 rpcz_store.cc:276] 0426 13:19:25.826883 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.827422 (+   539us) service_pool.cc:225] Handling call
0426 13:19:26.912008 (+1084586us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:26.912142 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":794898688}
I20240426 13:19:26.913640 26721 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:26.913990 26721 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:26.914739 26721 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.915791 26721 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.926823 26745 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.882256 26684 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.885078 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.929899 25104 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.865331 26664 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.931973 26664 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.840437 26600 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.932832 25025 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:26.905534 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.906944 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.934036 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.899209 24920 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.934926 24920 leader_election.cc:401] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.935382 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1268) took 1623 ms. Trace:
I20240426 13:19:26.937559 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:26.938402 26600 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.938889 26600 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:26.908630 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1611) took 1231 ms. Trace:
I20240426 13:19:26.941427 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.941972 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.910115 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.942816 24921 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.890352 25618 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:26.917934 25026 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.946610 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.918112 26717 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.918336 26756 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.918601 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.880309 26747 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.948026 26600 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.949343 26707 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.904721 26785 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:26.921855 26798 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20240426 13:19:26.922617 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46584.2us
W20240426 13:19:26.947917 26750 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.834s	user 0.000s	sys 0.005s
I20240426 13:19:26.954080 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.955627 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.957764 26747 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.959321 26785 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.962997 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.923624 26654 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.924302 24851 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 287) took 4426 ms. Trace:
I20240426 13:19:26.895375 26720 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.925217 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:26.927960 25022 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.881950 26771 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.943828 24951 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 13:19:26.940276 25023 rpcz_store.cc:276] 0426 13:19:25.676750 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.677803 (+  1053us) service_pool.cc:225] Handling call
0426 13:19:25.678063 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:26.536006 (+857943us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:26.908448 (+372442us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:26.908585 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1967849216}
I20240426 13:19:26.935719 25096 rpcz_store.cc:276] 0426 13:19:25.312117 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.314271 (+  2154us) service_pool.cc:225] Handling call
0426 13:19:26.935224 (+1620953us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540001ccd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:26.935331 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2818563456}
I20240426 13:19:26.971083 25620 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:26.972098 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:26.945165 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66351.4us
I20240426 13:19:26.945533 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.916960 26775 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.950627 25070 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:26.951870 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.974438 25070 leader_election.cc:401] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.950155 26718 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.969575 25030 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 4. Ops: []
I20240426 13:19:26.949901 26707 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.966979 24851 rpcz_store.cc:276] 0426 13:19:22.497452 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.497630 (+   178us) service_pool.cc:225] Handling call
0426 13:19:26.924197 (+4426567us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20240426 13:19:26.953580 26691 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.970497 24853 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:26.983017 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.973711 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.967876 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:26.975098 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54243.1us
I20240426 13:19:26.976213 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.984562 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.964530 26747 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.988510 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.989123 26291 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:26.993500 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.995514 26291 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:26.992869 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.995908 26291 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:26.999631 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.000350 25023 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:26.997675 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.001574 25103 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:26.982347 26720 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:26.962565 26774 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:27.002182 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.003230 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.005225 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.978183 26771 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:27.003532 26291 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.002604 25620 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:27.006659 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.991384 26553 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.015291 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.013723 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.986305 26691 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.013305 25096 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.012077 26291 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.991554 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:27.017999 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.019678 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.020983 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.025890 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.027473 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.008983 25616 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:26.984839 26679 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.030931 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.008306 26689 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.987532 26775 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 13:19:26.991875 26713 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240426 13:19:27.031724 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.039484 24952 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.011451 26507 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.041323 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:27.018213 26795 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.996577 26617 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20240426 13:19:27.022188 26691 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.995218 26664 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240426 13:19:27.024263 26413 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.048169 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:27.029644 26553 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.007064 26747 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20240426 13:19:27.041168 26588 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.031425 26551 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20240426 13:19:27.055536 26727 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.056747 26799 ts_tablet_manager.cc:902] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.031955 26690 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.037962 26696 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.042615 26772 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.011581 26610 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240426 13:19:27.019212 26707 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.071118 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1602) took 1513 ms. Trace:
W20240426 13:19:27.080987 26794 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20240426 13:19:27.089270 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1294) took 1099 ms. Trace:
I20240426 13:19:27.111176 25098 rpcz_store.cc:276] 0426 13:19:25.989565 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.989701 (+   136us) service_pool.cc:225] Handling call
0426 13:19:27.089073 (+1099372us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:27.089212 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2063662464}
I20240426 13:19:27.089517 25097 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.100000 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.100366 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.099452 26413 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:27.051801 26771 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.114025 26413 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.100072 26601 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.090480 26610 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.094401 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:27.124603 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 92968.4us
I20240426 13:19:27.113301 25103 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:27.126931 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.127010 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.127702 25103 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:27.129084 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.129402 25096 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.129774 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.100767 26720 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.130373 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.132289 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:27.124564 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95246.2us
I20240426 13:19:27.133162 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.134212 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:27.124536 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98784.2us
I20240426 13:19:27.135363 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.102844 25031 rpcz_store.cc:276] 0426 13:19:25.557361 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.557523 (+   162us) service_pool.cc:225] Handling call
0426 13:19:25.585195 (+ 27672us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.585513 (+   318us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.585542 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.595605 (+ 10063us) log.cc:842] Serialized 26 byte log entry
0426 13:19:25.765641 (+170036us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:25.765693 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.765777 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.947222 (+1181445us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:27.044378 (+ 97156us) raft_consensus.cc:1671] finished
0426 13:19:27.044399 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:27.071037 (+ 26638us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1804,"raft.queue_time_us":305,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11330,"spinlock_wait_cycles":7275264,"thread_start_us":123873,"threads_started":1,"wal-append.queue_time_us":493965}
I20240426 13:19:27.089125 26679 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.137516 25022 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:27.137209 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.140427 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.143797 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.145342 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.145473 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.147221 26664 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.148794 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.120525 26601 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.149633 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.148316 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.148277 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.123844 26610 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.126932 26413 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:27.161407 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144073us
I20240426 13:19:27.161736 26413 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.101795 26756 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240426 13:19:27.162050 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.163404 26777 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.162963 26748 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:27.136693 26720 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.138607 26713 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.158872 26610 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.159708 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.160317 26664 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.175496 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:27.154235 26551 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.158017 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.137820 26727 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.165205 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.135833 26747 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:27.171209 26699 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.172780 25620 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.148577 26696 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.175134 26291 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.160097 26601 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.154429 26772 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:27.158357 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.157546 26505 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.178052 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:27.199594 25100 raft_consensus.cc:595] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.158192 26679 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.182646 26481 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:27.157867 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.186870 25623 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.201171 26481 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:27.187741 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.201586 26481 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.202930 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.204357 26736 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.204367 26795 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.204799 26716 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:27.188900 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.192338 26551 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 13:19:27.194882 26800 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.146s	user 0.005s	sys 0.000s
I20240426 13:19:27.210723 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:27.212594 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75606.1us
W20240426 13:19:27.213670 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55436.1us
I20240426 13:19:27.214334 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.216346 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.217491 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.199183 26630 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.223006 26630 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.199422 25618 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.225315 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.226348 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.227288 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.233314 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.234560 26291 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.195312 26684 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.239369 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.209638 26481 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.265398 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107600us
I20240426 13:19:27.281791 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.282930 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.239369 26789 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.269179 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.200537 26772 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.198406 26747 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.283921 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.284570 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.289659 25100 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:27.290935 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 80299.4us
I20240426 13:19:27.263947 25104 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:27.240070 26505 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.271262 26601 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.223817 26630 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.342818 26630 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.342001 26791 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.235138 26696 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.235173 26610 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:27.225011 26551 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.198333 26777 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20240426 13:19:27.360251 26748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:47158 (request call id 6) took 5583 ms. Trace:
I20240426 13:19:27.271075 26681 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:27.296207 25030 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.385119 25100 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:27.344967 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 53051.4us
I20240426 13:19:27.224617 26699 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.266546 26625 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.270620 26688 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.272063 26756 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.290863 26776 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:27.295473 26751 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:19:27.295390 26737 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.292801 26775 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.292448 26772 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:27.360208 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:27.347859 26635 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.235002 26720 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.365429 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96330.5us
I20240426 13:19:27.365686 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.367885 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.368866 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.370270 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.380635 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:27.344439 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60597.9us
I20240426 13:19:27.353899 26698 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:19:27.512666 26698 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:27.513051 26698 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.513908 26698 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.515208 26775 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:27.387730 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.274109 26666 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.414654 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.415025 26774 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:27.517498 25028 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.385861 26777 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.422582 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:27.518724 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96259.7us
I20240426 13:19:27.521642 26688 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:27.522161 26688 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.474511 26776 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.522967 26776 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:27.524412 25023 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:27.526628 25028 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.526664 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.468963 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.438124 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.528226 24954 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.528615 24954 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:27.532747 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:27.471549 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20240426 13:19:27.518115 26777 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.542424 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:27.523016 26688 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.529503 24954 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.564841 24954 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:27.548863 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.548846 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.435366 26738 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.551821 25030 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 5
I20240426 13:19:27.426860 26689 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:27.570571 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:27.570574 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.527654 25031 raft_consensus.cc:2362] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4. Re-sending same reply.
I20240426 13:19:27.425544 26700 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:27.544474 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.543998 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.573915 25030 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240426 13:19:27.692850 25030 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:27.511821 26791 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.578054 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:27.671315 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 170616us
W20240426 13:19:27.575850 26761 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.144s	user 0.003s	sys 0.000s
I20240426 13:19:27.576716 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.723312 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.408314 26747 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.709465 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 136771us
I20240426 13:19:27.571345 26738 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.721956 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:27.724182 24948 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:27.710424 24951 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.736346 24947 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:27.736969 24947 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.593417 26481 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:27.507090 26718 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.521s	user 0.013s	sys 0.000s
W20240426 13:19:27.716975 26664 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.537s	user 0.009s	sys 0.001s
I20240426 13:19:27.743659 24954 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:27.745012 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.713291 26600 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.567833 26691 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.734330 26711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.035s	sys 0.044s
I20240426 13:19:27.789651 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.713824 26700 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.763399 26718 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:27.398900 26748 rpcz_store.cc:276] 0426 13:19:21.771338 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.771510 (+   172us) service_pool.cc:225] Handling call
0426 13:19:21.891393 (+119883us) log.cc:842] Serialized 492 byte log entry
0426 13:19:27.354744 (+5463351us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116470,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":5469,"thread_start_us":1607616,"threads_started":2,"wal-append.queue_time_us":396709,"wal-append.run_cpu_time_us":1887,"wal-append.run_wall_time_us":1001913}
I20240426 13:19:27.756006 25624 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.775892 26664 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:27.787211 26786 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.898s	user 0.014s	sys 0.001s
W20240426 13:19:27.777338 26711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.037s	sys 0.046s
W20240426 13:19:27.810618 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65731.9us
W20240426 13:19:27.809566 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276831us
I20240426 13:19:27.831161 24956 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:27.814862 26717 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.826289 26610 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:27.845350 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.792172 26738 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.849751 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.839658 26775 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:27.857255 24901 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:27.866320 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.866878 25102 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.854926 26635 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.855432 26717 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.870186 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 478) took 1185 ms. Trace:
I20240426 13:19:27.876121 25623 rpcz_store.cc:276] 0426 13:19:26.684408 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.684564 (+   156us) service_pool.cc:225] Handling call
0426 13:19:26.917279 (+232715us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:27.201566 (+284287us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:27.427733 (+226167us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:27.870002 (+442269us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400491950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:27.870123 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1154048768}
I20240426 13:19:27.882295 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.847551 26610 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.892776 24947 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.894647 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:27.898924 25105 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.899641 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.886451 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.899653 24947 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:27.908164 26748 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:27.908756 26748 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.845878 26816 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240426 13:19:27.878283 26691 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.862895 26775 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.886363 26823 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:27.858640 26773 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:27.924707 25030 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 13:19:27.930327 25030 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.874847 26684 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20240426 13:19:27.933701 25031 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:27.910526 26717 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.933758 25028 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.864354 26817 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20240426 13:19:27.935899 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.941560 25025 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:27.941807 25102 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:27.942533 25030 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.944140 25102 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.895545 26635 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240426 13:19:27.909694 26748 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.922699 26700 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:27.953625 25105 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.916476 26562 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20240426 13:19:27.919087 26791 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240426 13:19:27.934514 26823 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.959360 26819 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20240426 13:19:27.965623 25102 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.939074 26625 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240426 13:19:27.968757 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.972162 25102 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.978372 25105 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.973559 26823 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.965449 26600 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:27.965514 26684 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:27.990244 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72726.4us
I20240426 13:19:27.990424 25102 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:27.982571 26817 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.012382 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:28.013891 26822 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:19:27.979832 26617 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240426 13:19:28.011260 26776 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:28.023675 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 474904us
I20240426 13:19:27.998920 26816 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:28.021471 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.026067 25028 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:28.026592 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.001416 26610 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.999650 26817 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.036474 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.037112 25025 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:28.036758 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.989907 26684 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.039031 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.040858 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
W20240426 13:19:28.042167 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 120716us
I20240426 13:19:28.042955 26833 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.987818 26600 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.040349 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.024724 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.030865 26617 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 13:19:28.028934 26816 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.050019 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.029126 26700 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:28.050647 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.085603 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.085974 24947 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.088006 26707 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.041648 26610 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.029788 26799 tablet_copy_client.cc:308] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:28.039278 26775 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.097126 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:28.098299 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:28.095999 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45430.7us
I20240426 13:19:28.097739 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.102191 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.084606 26625 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.168443 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.171217 25029 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:28.095515 26700 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:28.176740 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70315.2us
I20240426 13:19:28.093938 26691 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:28.103943 26750 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20240426 13:19:28.185492 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83395.9us
I20240426 13:19:28.113533 26635 consensus_queue.cc:1225] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:19:28.095299 26825 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:28.094537 26828 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:28.101994 26757 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.108338 26839 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.175787 26744 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:28.186480 26813 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20240426 13:19:28.095891 26774 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20240426 13:19:28.185364 26691 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.185176 26503 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.196009 26825 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.177326 26708 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20240426 13:19:28.206341 26837 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20240426 13:19:28.210578 25030 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.212287 26681 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.198221 26551 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:28.177968 26816 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.197077 26833 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:28.222029 26630 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.879s	user 0.005s	sys 0.007s
I20240426 13:19:28.193276 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.223245 24951 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.227066 26691 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.215747 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 201) took 1244 ms. Trace:
I20240426 13:19:28.255622 24956 rpcz_store.cc:276] 0426 13:19:26.971607 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.971900 (+   293us) service_pool.cc:225] Handling call
0426 13:19:28.215540 (+1243640us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:28.215674 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1897286016}
I20240426 13:19:28.227820 26503 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.243211 24951 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.223331 26684 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.203105 26828 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.236465 25097 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.098834 26817 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.231812 26757 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:28.232575 26825 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.260798 25100 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.261282 26503 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.266435 25022 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 13:19:28.269129 26816 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.271168 25101 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:28.246717 26747 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:28.277052 26829 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:28.237722 26681 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.258745 24951 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.277395 26698 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.279495 26840 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20240426 13:19:28.291021 26757 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.294921 24949 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240426 13:19:28.277853 26776 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.278308 26849 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20240426 13:19:28.305774 24949 pending_rounds.cc:76] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:28.322731 24949 pending_rounds.cc:97] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 13:19:28.323087 24949 raft_consensus.cc:2866] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:28.300881 24951 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.301400 26508 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20240426 13:19:28.302309 26707 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:28.297427 26829 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.331532 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:28.333215 26832 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20240426 13:19:28.334407 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.335484 24956 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:28.324596 26772 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.828s	user 0.012s	sys 0.000s
I20240426 13:19:28.339538 26744 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:28.330091 26600 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.354334 24951 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:28.356316 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 181748us
I20240426 13:19:28.301662 26691 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.353170 24953 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:28.359078 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.361112 26700 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.362920 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.354962 26839 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:28.359917 26562 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240426 13:19:28.363663 26848 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.361176 26600 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:28.324697 26630 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 13:19:28.376057 26600 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:28.378198 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 272044us
I20240426 13:19:28.373971 26610 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.374541 26839 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.331488 26707 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.293646 26828 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.349584 26772 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.367312 26718 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.381747 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.380009 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.363324 26507 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.410108 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.364005 26789 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.367131 26857 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
W20240426 13:19:28.418138 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 231705us
I20240426 13:19:28.422040 25099 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:28.423556 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.425081 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 11fad756437243d2959f5998d6bf8309 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:28.376899 26600 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.428022 26855 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.436808 26855 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.437357 26601 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.438755 26855 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.439271 26855 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:28.440047 25098 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.428249 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 11fad756437243d2959f5998d6bf8309 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309
I20240426 13:19:28.443656 26829 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.446203 26617 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.446730 26617 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.410969 26718 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.416698 26774 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:28.417624 26789 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:28.417137 26507 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.446360 26839 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.453354 26851 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:28.477260 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119618us
W20240426 13:19:28.484874 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46310.4us
I20240426 13:19:28.485404 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.473840 26696 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.502476 26507 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.508165 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:28.517494 25104 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.517971 25104 raft_consensus.cc:603] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:28.457309 26681 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.531494 26707 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.532018 25099 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.483891 26851 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.533740 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:28.535461 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:28.536084 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:28.418957 26744 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.548161 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.515302 26617 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.553895 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.560422 25024 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.498279 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.556377 26773 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.558257 26858 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.559082 26828 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.550954 26691 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:28.581933 26863 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:19:28.566113 26825 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.585356 25028 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.571143 25104 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240426 13:19:28.520545 26789 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.526211 26696 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.498673 26718 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.579267 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.627405 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1598) took 1086 ms. Trace:
I20240426 13:19:28.590688 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 186) took 1223 ms. Trace:
I20240426 13:19:28.619402 25099 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:28.638283 25022 rpcz_store.cc:276] 0426 13:19:27.367364 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.367550 (+   186us) service_pool.cc:225] Handling call
0426 13:19:28.590502 (+1222952us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:28.590620 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1738735360}
I20240426 13:19:28.590083 25618 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.620921 26684 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.615820 26826 raft_consensus.cc:979] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:28.654134 26684 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:28.592777 26720 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.636816 24949 rpcz_store.cc:276] 0426 13:19:27.541004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.732154 (+191150us) service_pool.cc:225] Handling call
0426 13:19:27.732541 (+   387us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.323469 (+590928us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:28.627112 (+303643us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:28.627331 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1292505856}
I20240426 13:19:28.603412 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.658916 25028 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.654366 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.570719 26773 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.655370 26684 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.665951 26684 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.589905 26825 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.668489 26765 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240426 13:19:28.577163 26772 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.678814 26688 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:28.681293 26689 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.680680 26777 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.598966 26817 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.685633 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.686998 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.687606 24952 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:28.688613 24952 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:28.689327 26507 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.689679 24948 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:28.692713 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.692869 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.693332 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.693806 25618 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240426 13:19:28.693925 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:28.694006 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.661868 26630 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 13:19:28.694607 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:28.695844 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.698103 25101 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.591724 25041 tablet_copy_source_session.cc:205] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:28.681763 26689 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.704807 26773 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.699995 25028 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:28.701478 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.712213 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.620069 26681 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.715531 26773 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.717049 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1332) took 1329 ms. Trace:
I20240426 13:19:28.621186 26851 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.713670 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:28.719713 25098 rpcz_store.cc:276] 0426 13:19:27.387561 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.387701 (+   140us) service_pool.cc:225] Handling call
0426 13:19:27.388004 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.440457 (+1052453us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:28.716779 (+276322us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000ea454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:28.716982 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2393130240}
W20240426 13:19:28.725032 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 265330us
W20240426 13:19:28.728338 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 315884us
W20240426 13:19:28.739559 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 134180us
I20240426 13:19:28.714174 26718 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.743166 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.746526 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.750770 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.752280 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.754801 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.758971 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:28.759306 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.761940 26850 raft_consensus.cc:1067] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:28.763739 25022 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:28.767426 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.768260 25028 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.729141 26617 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:28.770900 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.773198 25104 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.737624 26481 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:28.743069 26738 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.782626 25025 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20240426 13:19:28.779227 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53282.3us
I20240426 13:19:28.784745 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.785804 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.786993 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:28.790421 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.706s (SENT) [suppressed 8 similar messages]
I20240426 13:19:28.792507 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:28.794377 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50580.3us
I20240426 13:19:28.796468 26851 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:28.797513 26851 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:28.755681 26711 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.801301 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.802186 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.775914 26664 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:28.805109 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.805843 25024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:28.820657 25024 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:28.821043 25024 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.774499 26617 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:28.808321 26721 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.808566 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.810120 26826 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.810717 26772 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.812525 25097 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:28.778280 26481 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.801636 26653 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:28.743461 26816 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240426 13:19:28.777997 26756 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:28.807801 26789 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.598249 26858 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.752157 26666 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 13:19:28.756866 26700 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240426 13:19:28.781417 26505 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20240426 13:19:28.831634 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:28.769765 26551 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240426 13:19:28.823906 26791 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20240426 13:19:28.912578 26625 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.823696 26717 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
W20240426 13:19:28.912169 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 499599us
I20240426 13:19:28.833110 26617 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.915742 26772 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.807804 26441 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20240426 13:19:28.795095 26757 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20240426 13:19:28.857023 25105 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:28.833777 26721 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.890267 26481 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.930373 26505 consensus_queue.cc:1225] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20240426 13:19:28.937281 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 104241us
W20240426 13:19:28.888932 24953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.019s	sys 0.065s
I20240426 13:19:28.823289 26711 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.910146 26775 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.928921 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.914952 26698 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.962495 26698 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.806106 26840 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20240426 13:19:28.912251 26826 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.967133 26698 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.929769 26717 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.934000 26861 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.922446 26756 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.972433 26756 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.975267 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.976442 25028 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:28.943909 24953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.020s	sys 0.065s
I20240426 13:19:28.930305 26696 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.961016 26865 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.946735 25024 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.961980 26711 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.987113 25023 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:28.987540 25069 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:28.992655 25024 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.017031 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.022459 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:29.023142 25105 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:29.023459 25105 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:29.024187 25105 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.025233 25105 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.026516 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.030876 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:28.938908 26707 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:28.984642 26708 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:29.035800 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.950335 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.934700 26625 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.035317 26503 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:29.043109 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.004s	sys 0.016s
W20240426 13:19:29.043746 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.004s	sys 0.016s
W20240426 13:19:29.044252 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 524291us
I20240426 13:19:29.046782 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.055436 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:29.034719 26776 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:29.041599 26707 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.044708 26507 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.040131 26551 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.017269 26696 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:28.928743 26852 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20240426 13:19:29.037410 26505 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.060593 24949 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.068384 24950 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.996384 26775 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.049319 26858 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.065155 26707 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:29.073781 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.075184 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.077275 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:29.084836 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.051191 26441 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.121927 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.062222 26666 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.025568 26721 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.125002 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 370322us
I20240426 13:19:29.042066 26711 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:19:29.087103 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.070693 26698 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.134429 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:29.079154 26600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.017s	sys 0.005s
I20240426 13:19:29.120265 26505 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.080008 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.148499 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.138211 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.166484 24952 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.165395 25623 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:29.060765 26826 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.144656 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.085805 26610 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:29.171754 24950 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.130667 26617 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.081571 26551 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.079428 26507 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.025751 26865 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.148308 26721 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:29.165387 26600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.017s	sys 0.009s
I20240426 13:19:29.062491 26772 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.192039 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.190260 25105 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:29.175344 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.056s	sys 0.157s
W20240426 13:19:29.184976 26684 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.000s
W20240426 13:19:29.191967 26744 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.011s	sys 0.023s
I20240426 13:19:29.203680 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:29.202383 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.056s	sys 0.158s
I20240426 13:19:29.157249 26858 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20240426 13:19:29.216637 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:29.219023 26689 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.001s
W20240426 13:19:29.209252 26744 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.012s	sys 0.023s
I20240426 13:19:29.175920 26698 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.216282 26817 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.000s
I20240426 13:19:29.125838 26441 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.189045 26610 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.210891 26630 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.140048 26666 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.215579 26748 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.236119 26505 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.236409 25099 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:29.236749 26625 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.219931 26684 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:29.219944 26681 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.221810 26562 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.238876 26869 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.226758 26711 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.210144 26503 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20240426 13:19:29.219471 26689 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:29.240484 26610 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:29.259032 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 468514us
I20240426 13:19:29.241150 26551 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.238202 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:29.251758 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.020s	sys 0.021s
I20240426 13:19:29.252360 26562 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.290573 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.020s	sys 0.022s
I20240426 13:19:29.252506 26684 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:29.292677 25104 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:29.238127 26630 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.302104 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 347225us
I20240426 13:19:29.240430 26748 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.316691 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.319518 25099 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.319913 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.251176 26681 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.326709 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.328521 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.322304 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.323791 25101 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:29.329874 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:29.292155 26775 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:29.325644 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3fe8f3015c01480fb67b39d3f5a947c7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.333508 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3fe8f3015c01480fb67b39d3f5a947c7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.325840 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.331596 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.332504 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.319301 26505 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.346668 26481 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.348225 26551 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.348981 26630 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.348836 26551 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.350536 26748 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.315542 26617 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.350975 26861 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.328572 25617 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:29.351284 26684 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.355515 24950 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:29.361541 25024 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:29.362030 26698 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.350920 26551 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.348992 26689 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.348256 26507 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.362819 26562 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.364176 26441 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:29.329274 26720 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.000s
I20240426 13:19:29.332100 26600 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.263746 26869 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.395224 26775 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.395790 26775 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.396309 26869 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.397303 26699 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:29.397727 26699 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:29.394134 25099 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:29.329262 26817 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.401293 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.402385 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.402508 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.404003 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.404459 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.413443 25101 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:29.416021 25031 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:29.422339 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:29.388949 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.410283 26825 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.412760 26698 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:29.430738 26718 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.678s	user 0.010s	sys 0.000s
I20240426 13:19:29.427124 26817 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20240426 13:19:29.470247 26873 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20240426 13:19:29.401651 26600 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.461410 26707 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.473306 26600 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.474030 26707 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.475479 24948 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:29.461489 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:29.452023 25025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.007s	sys 0.016s
W20240426 13:19:29.477124 25025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.008s	sys 0.016s
I20240426 13:19:29.490377 24955 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:19:29.490882 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 265026us
I20240426 13:19:29.493811 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:29.466465 26882 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:29.503906 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.510669 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.511760 25024 raft_consensus.cc:2396] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 13:19:29.495637 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.516012 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:29.537344 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2d6f5c5283464089ad1bc18c13b427cc, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.537866 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2d6f5c5283464089ad1bc18c13b427cc, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.557132 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.506341 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:29.563280 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:29.563979 25101 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
W20240426 13:19:29.564986 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52415.5us
I20240426 13:19:29.495484 26441 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:29.565516 24953 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.032s	sys 0.049s
W20240426 13:19:29.573148 24953 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.032s	sys 0.049s
I20240426 13:19:29.499298 26505 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:29.578282 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1304) took 2014 ms. Trace:
I20240426 13:19:29.499673 26481 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.501545 26413 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:29.468506 26698 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.560006 26789 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.622s	user 0.009s	sys 0.001s
I20240426 13:19:29.594564 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.465412 26825 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.604673 25099 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.594686 24953 rpcz_store.cc:276] 0426 13:19:27.563820 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.727674 (+163854us) service_pool.cc:225] Handling call
0426 13:19:28.352716 (+625042us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:28.352908 (+   192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.353079 (+   171us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:28.384310 (+ 31231us) log.cc:842] Serialized 14 byte log entry
0426 13:19:29.577877 (+1193567us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:29.577937 (+    60us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:29.577993 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.578070 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:29.578205 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18047,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":21683,"spinlock_wait_cycles":5111680,"thread_start_us":1193178,"threads_started":2,"wal-append.queue_time_us":455874}
I20240426 13:19:29.567898 26707 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.601119 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240426 13:19:29.537504 26654 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:29.601583 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 390531us
I20240426 13:19:29.506155 26882 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.562202 26551 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.558843 26718 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.572945 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:29.667611 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94742.4us
I20240426 13:19:29.565091 26877 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.653523 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:29.572335 26817 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.600121 26505 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.664995 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.671885 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:29.672266 25103 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:29.627985 26870 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20240426 13:19:29.577670 26765 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20240426 13:19:29.668623 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:29.678143 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 358996us
W20240426 13:19:29.652529 26773 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.929s	user 0.010s	sys 0.002s
I20240426 13:19:29.602156 26441 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.600935 26413 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:29.601431 26775 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:29.683683 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 476149us
I20240426 13:19:29.645768 26681 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.697763 24953 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:29.700960 24953 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:29.698172 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:29.703140 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 486666us
I20240426 13:19:29.666867 26882 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.703907 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:29.704151 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.707520 25099 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.699477 25101 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.671512 26789 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:29.674669 26505 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.708954 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.710114 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.695613 25026 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20240426 13:19:29.714653 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:29.714962 24956 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.019s	sys 0.041s
I20240426 13:19:29.717528 24947 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:29.699064 26756 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.687633 26748 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:29.719147 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.664177 26707 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.720643 25623 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:29.721671 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.702180 24953 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.722280 24956 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.019s	sys 0.041s
I20240426 13:19:29.739552 24953 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.705523 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:29.668100 26654 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.707315 26508 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.760342 26508 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.760784 26698 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.761379 26654 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.762158 26508 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.671160 26551 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.697643 26441 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.764138 26551 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.764747 26441 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.765650 24950 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:29.770756 26707 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.772174 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:29.774168 26738 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.979s	user 0.010s	sys 0.000s
I20240426 13:19:29.774725 25102 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:29.711827 26789 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.748068 26681 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:29.717036 26773 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.757056 26711 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:29.723394 26481 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:29.777699 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.778205 24953 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240426 13:19:29.795248 24953 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:29.795692 24953 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 13:19:29.796002 24953 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:29.796363 24953 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:29.724805 26887 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.796660 24953 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:19:29.897778 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 229026us
I20240426 13:19:29.807861 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:29.903000 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 95040.6us
I20240426 13:19:29.904750 25026 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240426 13:19:29.908507 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:29.800829 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62917.8us
I20240426 13:19:29.912197 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.915936 25028 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:29.915799 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.918331 24953 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:29.791373 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.926110 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.711437 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.740356 26413 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.797719 26481 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.787312 26711 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:29.957345 26481 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.958938 26698 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.958798 26711 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.959992 26698 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.780313 26713 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.963362 26713 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.753782 26717 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.965641 26890 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.966789 26903 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:29.966691 26890 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.727885 26748 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.741187 26756 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.968669 26748 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:29.970578 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.970942 26756 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.807575 26765 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.972129 26765 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.974150 26698 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.974718 26765 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.789324 26789 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.967756 26903 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.978451 26903 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.979048 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.730226 26861 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20240426 13:19:29.990140 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.990195 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.991158 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1666) took 1164 ms. Trace:
I20240426 13:19:29.673435 26877 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:29.848340 26883 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.160s	user 0.001s	sys 0.001s
I20240426 13:19:29.867363 26882 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.999423 25097 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:29.775264 26884 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.001328 26884 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.003010 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.004501 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 201) took 1212 ms. Trace:
W20240426 13:19:29.852392 26885 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20240426 13:19:30.005079 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 204) took 1201 ms. Trace:
I20240426 13:19:30.005688 25022 rpcz_store.cc:276] 0426 13:19:28.803417 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.803536 (+   119us) service_pool.cc:225] Handling call
0426 13:19:28.811378 (+  7842us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.829394 (+ 18016us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.829435 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:28.829488 (+    53us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:28.829531 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.004565 (+1175034us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.004659 (+    94us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.004870 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40914816}
I20240426 13:19:30.006862 26877 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:30.006984 26912 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.006911 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.942276 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:30.008584 26600 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:29.951712 26684 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:29.955099 26689 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.007740 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.025684 25622 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:30.072191 25617 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:29.739529 26829 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20240426 13:19:29.980579 26888 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.009560 26884 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.992427 25023 rpcz_store.cc:276] 0426 13:19:28.826394 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.826616 (+   222us) service_pool.cc:225] Handling call
0426 13:19:28.827126 (+   510us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.994741 (+167615us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.990663 (+995922us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:29.990945 (+   282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366953856}
I20240426 13:19:29.982177 26681 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.987916 26906 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.989118 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.989833 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.991521 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:29.811585 26887 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.086266 26887 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.009223 26600 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.086450 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:30.026856 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 449) took 1928 ms (client timeout 2492 ms). Trace:
W20240426 13:19:30.087608 25624 rpcz_store.cc:269] 0426 13:19:28.098181 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.098817 (+   636us) service_pool.cc:225] Handling call
0426 13:19:30.026397 (+1927580us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.026659 (+   262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2241880960}
I20240426 13:19:30.088523 26600 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.089790 26888 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.091084 24949 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.093888 26906 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.083534 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.095830 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.081473 26738 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:30.098268 26738 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:30.099560 26738 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.100847 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.102583 26906 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.004148 25097 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:30.010221 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 202) took 1218 ms. Trace:
I20240426 13:19:29.981021 26718 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.104084 26689 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.108924 26689 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.113996 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.062045 26882 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.103848 26888 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.004949 25025 rpcz_store.cc:276] 0426 13:19:28.791646 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.793555 (+  1909us) service_pool.cc:225] Handling call
0426 13:19:28.828139 (+ 34584us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.828476 (+   337us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.828506 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.834172 (+  5666us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.477640 (+643468us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:29.477695 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.477782 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.004119 (+526337us) raft_consensus.cc:1671] finished
0426 13:19:30.004153 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.004340 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1908,"raft.queue_time_us":542,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":25825,"thread_start_us":643125,"threads_started":1,"wal-append.queue_time_us":794805}
I20240426 13:19:30.005106 25105 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:30.054049 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.140831 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:30.021849 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74786.2us
I20240426 13:19:30.021327 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.981854 26913 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:30.010458 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61346.2us
I20240426 13:19:30.013478 26912 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.054932 26654 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.097784 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.967607 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.109742 26777 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.009634 25028 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:30.012543 26707 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.146811 26707 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.147924 24948 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:30.149006 26707 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.121069 26601 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.152256 26654 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.121534 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.121752 26738 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.156312 26747 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:30.156718 26826 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.156803 26747 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.157483 26747 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:30.122217 26914 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.111s	user 0.002s	sys 0.000s
I20240426 13:19:30.154639 26505 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.122323 26900 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.161929 26505 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.162307 26900 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.122862 26890 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.123512 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1233) took 1365 ms. Trace:
I20240426 13:19:30.125890 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.165856 25097 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.165344 26913 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.166253 26900 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.104784 25029 rpcz_store.cc:276] 0426 13:19:28.792002 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.792179 (+   177us) service_pool.cc:225] Handling call
0426 13:19:30.009797 (+1217618us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400415fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.009978 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":917025536}
I20240426 13:19:30.167629 26865 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.126775 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.129926 26653 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:30.168398 26653 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.165366 25102 rpcz_store.cc:276] 0426 13:19:28.757842 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.759086 (+  1244us) service_pool.cc:225] Handling call
0426 13:19:30.123147 (+1364061us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.123336 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2334134528}
I20240426 13:19:30.169530 26653 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.170953 26653 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.174722 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.083281 26684 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.177343 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.177122 26684 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.179452 26666 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:30.131054 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 407266us
I20240426 13:19:30.182521 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.169150 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.184798 25620 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.186285 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.187768 26912 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:30.189220 26879 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.623s	user 0.003s	sys 0.000s
I20240426 13:19:30.188845 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.191805 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.192087 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.192605 25025 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.129647 26789 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.125926 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1356) took 1280 ms. Trace:
I20240426 13:19:30.139123 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:30.140996 26864 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:19:30.076593 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 452) took 1005 ms. Trace:
I20240426 13:19:30.197917 25104 rpcz_store.cc:276] 0426 13:19:28.845032 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.845233 (+   201us) service_pool.cc:225] Handling call
0426 13:19:28.917748 (+ 72515us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:29.302971 (+385223us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.125518 (+822547us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.125762 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":861890816}
I20240426 13:19:30.197069 26872 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.204865 26872 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.206154 26852 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.206867 26852 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.207427 26872 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.152923 26601 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.154973 25623 raft_consensus.cc:2354] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
W20240426 13:19:30.155473 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 205278us
I20240426 13:19:30.155627 26654 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.174911 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 271) took 6263 ms. Trace:
I20240426 13:19:30.182633 26799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44520 (request call id 290) took 3421 ms. Trace:
I20240426 13:19:30.132639 26689 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.134590 24949 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.142300 25099 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.201368 26562 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:30.201754 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 489) took 1741 ms (client timeout 1633 ms). Trace:
I20240426 13:19:30.202066 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 222) took 1692 ms. Trace:
I20240426 13:19:30.202734 26773 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.202080 26816 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.203403 26748 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.199697 25622 rpcz_store.cc:276] 0426 13:19:29.070854 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:29.070989 (+   135us) service_pool.cc:225] Handling call
0426 13:19:30.024973 (+953984us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:30.025312 (+   339us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:30.075581 (+ 50269us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.076541 (+   960us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24290048}
I20240426 13:19:30.203986 26696 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:30.207813 26871 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.733s	user 0.002s	sys 0.000s
I20240426 13:19:30.208711 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.210309 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.224166 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:30.216917 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 220) took 1714 ms (client timeout 1984 ms). Trace:
I20240426 13:19:30.217283 26562 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.220019 26878 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.752s	user 0.000s	sys 0.003s
W20240426 13:19:30.220135 25616 rpcz_store.cc:269] 0426 13:19:28.459785 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.462521 (+  2736us) service_pool.cc:225] Handling call
0426 13:19:30.201411 (+1738890us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.201565 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":836499712}
I20240426 13:19:30.222790 26707 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.223616 26601 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.167676 26913 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.262615 26913 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.263209 26913 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.265157 25098 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20240426 13:19:30.265336 26913 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.234937 26852 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.265898 26721 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.263867 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1353) took 1589 ms. Trace:
I20240426 13:19:30.234158 26773 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.266474 26721 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.236006 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ece98ef0f07f4184bedf2cb96b628f20, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:30.267827 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ece98ef0f07f4184bedf2cb96b628f20, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.231500 26816 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.266882 25100 rpcz_store.cc:276] 0426 13:19:28.674353 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.674534 (+   181us) service_pool.cc:225] Handling call
0426 13:19:28.720020 (+ 45486us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.754644 (+ 34624us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.754685 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.755164 (+   479us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.164846 (+409682us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:29.164905 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.164997 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.219097 (+1054100us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:30.263352 (+ 44255us) raft_consensus.cc:1671] finished
0426 13:19:30.263372 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.263699 (+   327us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34182,"raft.queue_time_us":36770,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":6613,"spinlock_wait_cycles":465287936,"thread_start_us":392119,"threads_started":1,"wal-append.queue_time_us":641359}
I20240426 13:19:30.268553 26816 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.240231 26696 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.241241 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.241231 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.241994 26789 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.242695 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.248623 25030 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:30.278345 26688 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.252537 26747 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.320168 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.320394 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.212785 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1611) took 1133 ms. Trace:
I20240426 13:19:30.321388 26747 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.265900 26825 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.324059 24948 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:30.324394 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.223887 26713 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.215654 26799 rpcz_store.cc:276] 0426 13:19:26.765217 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.765367 (+   150us) service_pool.cc:225] Handling call
0426 13:19:27.070272 (+304905us) ts_tablet_manager.cc:903] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
0426 13:19:29.296472 (+2226200us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:19:30.014282 (+717810us) tablet_metadata.cc:641] Metadata flushed
0426 13:19:30.014676 (+   394us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:19:30.181366 (+166690us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:19:30.181821 (+   455us) tablet_replica.cc:205] Creating consensus instance
0426 13:19:30.182267 (+   446us) ts_tablet_manager.cc:958] Replica registered.
0426 13:19:30.182364 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":25453,"tablet-copy.queue_time_us":173608,"thread_start_us":1014304,"threads_started":5}
I20240426 13:19:30.327142 26799 tablet_copy_client.cc:783] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:30.327652 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.258733 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.343845 26773 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.260982 26562 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.260888 26699 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.344507 24950 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:30.344699 25028 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:30.345142 26699 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.238603 26679 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.348356 26679 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.348861 25105 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:30.350407 26679 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:30.275993 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51363.1us
I20240426 13:19:30.351181 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.352169 25069 leader_election.cc:398] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
W20240426 13:19:30.256500 24955 rpcz_store.cc:269] 0426 13:19:28.500213 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.500396 (+   183us) service_pool.cc:225] Handling call
0426 13:19:30.214506 (+1714110us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.214648 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2165327104}
I20240426 13:19:30.352303 26816 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.215492 24951 rpcz_store.cc:276] 0426 13:19:28.509420 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.510927 (+  1507us) service_pool.cc:225] Handling call
0426 13:19:28.511303 (+   376us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.511908 (+   605us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.511948 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.512518 (+   570us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.291099 (+778581us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:29.291158 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.291259 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.188352 (+897093us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:30.201604 (+ 13252us) raft_consensus.cc:1671] finished
0426 13:19:30.201626 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.201878 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":165989760,"thread_start_us":778241,"threads_started":1,"wal-append.queue_time_us":963587}
I20240426 13:19:30.278831 26688 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.279021 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.285902 26852 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:30.357183 26852 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.357609 26852 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.357856 26681 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.358264 26681 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.358598 26681 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.286480 26656 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.287438 25617 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:30.288727 26916 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20240426 13:19:30.360213 25025 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.361074 26656 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.361462 26656 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.361820 26656 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.290067 26817 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:30.367707 26717 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.352455 25069 leader_election.cc:401] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:30.369776 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.370524 26713 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:30.370826 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.371071 26713 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.295024 26696 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:30.295603 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85400.8us
I20240426 13:19:30.296226 26903 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.297780 26721 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.298555 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.211783 24853 rpcz_store.cc:276] 0426 13:19:23.911498 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.038475 (+126977us) service_pool.cc:225] Handling call
0426 13:19:30.174697 (+6136222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:27.356877 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:27.378893 (+ 22016us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:27.378932 (+    39us) write_op.cc:427] Acquired schema lock
0426 13:19:27.378948 (+    16us) tablet.cc:660] Decoding operations
0426 13:19:27.379368 (+   420us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:27.379428 (+    60us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:27.379463 (+    35us) tablet.cc:683] Acquiring locks for 2 operations
0426 13:19:27.383094 (+  3631us) tablet.cc:699] Row locks acquired
0426 13:19:27.383115 (+    21us) write_op.cc:260] PREPARE: finished
0426 13:19:27.383273 (+   158us) write_op.cc:270] Start()
0426 13:19:27.383385 (+   112us) write_op.cc:275] Timestamp: P: 1714137567383222 usec, L: 0
0426 13:19:27.383405 (+    20us) op_driver.cc:331] REPLICATION: starting
0426 13:19:27.385603 (+  2198us) log.cc:842] Serialized 912 byte log entry
0426 13:19:28.101765 (+716162us) op_driver.cc:444] REPLICATION: finished
0426 13:19:28.452363 (+350598us) write_op.cc:300] APPLY: starting
0426 13:19:28.460895 (+  8532us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:28.464956 (+  4061us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:28.476831 (+ 11875us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:28.553719 (+ 76888us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:28.586949 (+ 33230us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0426 13:19:28.587025 (+    76us) write_op.cc:311] APPLY: finished
0426 13:19:28.587221 (+   196us) log.cc:842] Serialized 29 byte log entry
0426 13:19:28.587398 (+   177us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:28.587488 (+    90us) write_op.cc:446] Released schema lock
0426 13:19:28.587533 (+    45us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:28.587832 (+   299us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:28.587868 (+    36us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":855745,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":304000,"apply.run_cpu_time_us":2914,"apply.run_wall_time_us":160599,"num_ops":2,"prepare.queue_time_us":289666,"prepare.run_cpu_time_us":29185,"prepare.run_wall_time_us":328618,"raft.queue_time_us":262088,"raft.run_cpu_time_us":38601,"raft.run_wall_time_us":303119,"replication_time_us":711523,"thread_start_us":741837,"threads_started":4,"wal-append.queue_time_us":341091,"wal-append.run_cpu_time_us":30173,"wal-append.run_wall_time_us":1866685}]]}
I20240426 13:19:30.303309 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.304459 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.308099 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:30.309363 26867 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240426 13:19:30.374305 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75901.4us
I20240426 13:19:30.309787 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.310662 26839 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.310879 26748 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.313542 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.314486 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.318459 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.258410 25026 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.394372 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.321445 24954 rpcz_store.cc:276] 0426 13:19:29.074817 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:29.075913 (+  1096us) service_pool.cc:225] Handling call
0426 13:19:29.076259 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:29.204088 (+127829us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:29.204130 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:29.204170 (+    40us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:29.204200 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.198395 (+994195us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540004b354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.208135 (+  9740us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.208334 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293524352}
I20240426 13:19:30.394975 25623 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:30.328819 26413 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.279239 26481 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.397102 26481 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.397910 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.398908 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.399547 25024 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.399435 26481 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.400527 26481 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:30.335487 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 196) took 1914 ms. Trace:
I20240426 13:19:30.400878 26481 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.400991 25030 rpcz_store.cc:276] 0426 13:19:28.420646 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.420784 (+   138us) service_pool.cc:225] Handling call
0426 13:19:28.421133 (+   349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:30.278009 (+1856876us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.335153 (+ 57144us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54004169d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.335418 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4201729792}
I20240426 13:19:30.400909 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:30.401919 26481 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.392306 26852 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.405923 26852 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.336732 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1229) took 1580 ms. Trace:
I20240426 13:19:30.338461 26711 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.327826 26799 tablet_copy_client.cc:647] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
W20240426 13:19:30.356097 26707 consensus_queue.cc:1159] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 5 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:30.356648 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77750.6us
W20240426 13:19:30.288806 26880 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.790s	user 0.002s	sys 0.000s
I20240426 13:19:30.359781 26688 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.355367 26816 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.376368 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.375463 26825 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.415588 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.415925 26825 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.416345 26825 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.416648 26791 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.418224 26791 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.418951 26791 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:30.377086 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69219.1us
I20240426 13:19:30.376814 26839 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.420605 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.421669 26839 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.377501 26849 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.422317 26849 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.422591 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.422878 26859 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.355556 26893 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:19:30.423444 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.377945 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.379798 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.425086 25025 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.372747 26903 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.426086 26903 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.427196 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.375384 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.428290 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:30.380249 26747 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:30.428961 26747 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:30.430145 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53833.6us
I20240426 13:19:30.429841 26747 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.380988 25616 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:30.382622 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.432801 26679 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.433097 25624 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:30.434218 26688 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.434834 26688 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.383734 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.385017 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.386024 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.391515 26789 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.406484 25098 rpcz_store.cc:276] 0426 13:19:28.756563 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.758548 (+  1985us) service_pool.cc:225] Handling call
0426 13:19:30.264700 (+1506152us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:30.265603 (+   903us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.336529 (+ 70926us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54001b3350. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.336665 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3463254784}
I20240426 13:19:30.422713 26849 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.438314 26859 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.439083 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.423568 26893 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20240426 13:19:30.378638 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75441.5us
I20240426 13:19:30.436872 26688 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.380160 26656 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.419850 26481 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.450269 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.452495 25624 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:30.406499 26721 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.454002 26713 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:30.461028 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.470976 25027 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.461923 26791 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.435865 26825 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.476303 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.463469 26903 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.473546 25024 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.481204 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.484192 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.484282 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.370932 26681 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.487958 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72591.9us
W20240426 13:19:30.487932 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 72460.9us
I20240426 13:19:30.495865 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496022 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496254 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496829 24955 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.498653 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.498775 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.499217 24955 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:30.499855 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.471858 26839 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:30.500638 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.474021 26721 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.505718 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.478714 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.515403 24953 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.484817 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.518237 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.518592 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.522575 24948 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.522751 25097 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:30.522065 24951 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.519088 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.521463 25619 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:30.525367 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.525518 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.515375 25105 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:30.524645 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.528179 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.528791 25099 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:30.529603 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.510694 26721 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.532230 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.534314 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.532550 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.532907 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.534745 25024 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:30.505491 26713 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:30.506842 26839 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.514701 26748 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:30.531737 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.540207 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.540293 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.536307 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.538715 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.538192 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:30.543509 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:30.485342 26849 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.535225 25617 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.545859 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.545121 25025 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.549518 26858 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:30.549849 26894 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.550259 25101 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.550709 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.551308 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.547243 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.586215 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.551466 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.556507 25098 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:30.546537 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.557143 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.561796 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.577653 25103 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.563807 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.519520 26893 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20240426 13:19:30.547029 26839 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.591315 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.589285 24955 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:30.592288 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.592968 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.594000 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:30.597529 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.546133 26689 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.604480 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.608505 26816 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:30.609573 26900 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:30.605620 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.580255 26858 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:30.604282 26865 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.585965 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.590543 26748 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.582867 26738 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:30.600402 26848 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.624825 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.620971 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.544873 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.589850 26789 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:30.637830 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.638062 25099 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:30.638657 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.640748 25029 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.641809 25103 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:30.642539 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:30.628414 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 163413us
W20240426 13:19:30.650846 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56894.6us
I20240426 13:19:30.557508 26747 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.592988 26717 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.606381 26689 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.634243 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.651938 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.628053 26852 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.654018 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.645566 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.658277 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.635345 26848 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.661449 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.668103 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.665630 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.673142 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.666762 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.680151 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.681030 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.649997 26656 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.710109 25103 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.657258 26791 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.664011 26825 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.644268 26789 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.670805 25026 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.596202 26894 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.675109 26848 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.723094 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:30.793279 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54908us
I20240426 13:19:30.793989 25024 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:30.730448 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.794363 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.669539 26689 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.767424 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.746763 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.797722 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.798117 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:30.799043 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.799714 26699 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.800192 26699 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.800554 26699 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.801502 26894 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.802932 24951 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:30.804157 25102 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.805224 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.805938 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:30.808521 26849 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.810783 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.671418 26653 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.719302 26717 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.754244 26713 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.813066 25104 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.680688 26681 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.722683 26666 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.825866 26666 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.826260 26666 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.739373 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.739008 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.830148 25029 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
W20240426 13:19:30.833122 26921 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20240426 13:19:30.835988 26717 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.843863 25102 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.845889 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.738731 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.671774 26903 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.719892 25097 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.798919 26707 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.739064 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.716396 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.848345 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.848395 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.870774 24921 leader_election.cc:401] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240426 13:19:30.880856 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58413.3us
I20240426 13:19:30.680351 26776 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.884405 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.885219 26562 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.886360 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.853238 26748 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.888051 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.888723 24954 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:30.890426 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:30.891281 26681 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.891635 26681 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.891968 26681 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.892447 26911 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:30.892462 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 125185us
I20240426 13:19:30.893888 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.857367 25027 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:30.897732 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.851953 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:30.739426 26773 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.911540 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:30.912966 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114017us
I20240426 13:19:30.666023 26858 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.920287 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:30.712123 26826 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.926573 26848 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.928087 26824 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.929663 26757 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240426 13:19:30.930827 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 214528us
I20240426 13:19:30.968533 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.931298 26825 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.969420 26825 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.941876 26826 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.969854 26825 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.970054 26826 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.972545 26848 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:30.973865 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 234879us
I20240426 13:19:30.973920 26839 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.975162 24950 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.935708 26902 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:30.976558 26902 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.978277 26902 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.978896 26684 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.981555 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.982244 25029 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.950564 26656 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.984369 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.984544 26656 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.985008 26656 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.985172 25024 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.917924 26773 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.986251 26773 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.990562 24948 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240426 13:19:30.990929 24948 pending_rounds.cc:76] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:30.936163 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:30.937449 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 274715us
I20240426 13:19:30.952600 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.882201 26791 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.944360 24921 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.000159 24921 leader_election.cc:401] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:19:30.949156 26717 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.859797 26699 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.960862 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.018482 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 81062.7us
I20240426 13:19:31.018978 24954 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.944890 26688 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.964702 26413 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.965135 26849 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.965624 26894 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.904314 26776 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.966161 26872 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:31.025494 26852 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.967533 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.991132 24948 pending_rounds.cc:97] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.2
W20240426 13:19:31.024664 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72162.3us
I20240426 13:19:31.047816 24948 raft_consensus.cc:2866] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:30.984560 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:31.026041 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.048192 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.905079 26747 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.044374 26872 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.042132 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.044376 25030 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240426 13:19:31.026124 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.050274 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.028980 26824 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.051741 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67322us
I20240426 13:19:31.021873 26413 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.058239 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.052852 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.051901 25103 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.022677 26688 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.030716 26913 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:31.030468 26839 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.046193 26776 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.067101 24996 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:31.072284 24996 leader_election.cc:401] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:31.062909 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.052385 26900 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.052335 25617 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:31.058480 26872 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.060925 26684 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:31.029013 26859 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240426 13:19:31.057746 26867 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.081144 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.030265 26717 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.061146 26688 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.080291 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.061642 26413 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.076134 26601 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240426 13:19:31.079789 26684 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.051239 26894 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.085659 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.070796 26902 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.116304 26912 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.071709 26791 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:31.086886 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.097079 26824 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.100376 26562 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:31.084849 26867 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.105681 26681 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.107017 26699 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.107604 26684 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.108522 26894 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.071707 26911 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:31.101606 26666 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.156839 25029 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:31.150385 26721 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.570s	user 0.002s	sys 0.010s
W20240426 13:19:31.163452 26933 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20240426 13:19:31.151635 26867 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.166330 26711 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.166603 26653 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.167600 26707 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:31.148288 26562 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:31.187520 25030 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.164474 26791 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.172848 26913 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:31.176447 26789 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.170662 26894 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.210309 25619 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:31.207310 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 179763us
I20240426 13:19:31.207221 26825 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.179500 26721 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.194006 26791 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.203212 26717 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.231323 26930 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.232632 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.235272 26773 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.237268 26721 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.237803 25100 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.263262 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:31.263975 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.204210 26656 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.267572 26656 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.245499 26791 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.273641 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.256714 26562 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.278213 24853 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.257428 26747 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.229650 26894 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.283787 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.255060 26824 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.293190 26562 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.342113 26826 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.343883 25102 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.347153 26826 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.308140 26861 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.309749 26666 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.253993 26930 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.320281 26681 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.324694 26824 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.325179 26825 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.333938 26858 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.335332 26852 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.338874 26413 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.339663 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.341156 26773 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.340972 26872 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.279894 26776 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.272715 26859 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.306391 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:19:31.362579 24953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.000s	sys 0.049s
W20240426 13:19:31.353379 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.041s	sys 0.024s
W20240426 13:19:31.400337 24953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.000s	sys 0.050s
W20240426 13:19:31.400391 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.042s	sys 0.024s
I20240426 13:19:31.348055 26689 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.375504 26711 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.398944 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.410344 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.414223 25096 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:31.418224 24853 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.395769 26653 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.373760 26861 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.453203 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.414369 26711 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.454525 25619 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.426602 26653 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.466037 26688 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.469475 26861 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.426113 26684 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.472043 26684 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.470865 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.472064 26867 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.403059 26859 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.488943 26688 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.492827 26826 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.497257 26776 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.497655 26894 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.496960 26826 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.506242 25096 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:31.427058 26849 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20240426 13:19:31.508754 26689 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.509589 26872 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.510396 26413 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:31.517856 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 462491us
I20240426 13:19:31.510627 26826 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.463771 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.512614 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:31.519243 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 502842us
W20240426 13:19:31.520478 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 425520us
I20240426 13:19:31.523415 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.467952 26791 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.527763 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 440979us
I20240426 13:19:31.535137 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.536293 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.538976 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.541266 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.526459 26872 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:31.542991 25102 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.588829 26872 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:31.589713 26872 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.590926 26872 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.571044 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.595927 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.596441 25102 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:31.544147 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:31.545588 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 496115us
W20240426 13:19:31.533857 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 471046us
I20240426 13:19:31.548861 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.600459 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.599952 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.604048 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.604645 25027 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.549291 26867 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.605450 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.552366 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.544957 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.600484 25623 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.606258 25102 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:31.606704 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61825.1us
W20240426 13:19:31.606861 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62786.7us
I20240426 13:19:31.553799 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.530349 25617 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.554755 25100 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:31.542716 26656 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.555701 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.556193 26562 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.557065 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.613476 25103 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.558027 26913 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.559176 25026 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:19:31.559762 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.562741 26799 tablet_copy_client.cc:523] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:31.562629 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.535876 26894 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:31.566536 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:31.599095 26839 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.003s	sys 0.024s
I20240426 13:19:31.568166 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.569221 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.569578 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.571038 26942 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:19:31.621830 24948 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.571466 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.622470 25099 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.571584 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.574555 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.622987 24951 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.576480 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.577538 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.623703 26894 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.625909 26894 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.626494 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.627053 25100 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.619632 26852 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:31.632651 26852 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.633069 26852 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.604586 26900 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.634070 26900 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.635816 26900 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.612215 26666 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:31.636884 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.637857 25025 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.638752 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:31.639035 25069 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:31.640756 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.580214 26930 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.534147 26413 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:31.584124 26824 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.654743 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.655062 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.585508 26791 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.610849 26848 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.605494 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.537813 26938 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.543743 26717 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:31.607993 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.609966 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.611779 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:31.676522 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68612.1us
I20240426 13:19:31.614328 25098 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.677116 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.616726 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.676451 25029 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.677510 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.616623 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.612762 25104 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:31.617966 24954 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.568018 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.680538 24950 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.624022 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:31.578451 25619 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.643359 25100 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:31.622215 26707 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.643790 24951 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:31.608949 26935 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:31.703910 24952 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.642302 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:31.602814 26653 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.620136 26562 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.689113 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47467.4us
I20240426 13:19:31.689684 25096 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.657763 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.688686 26894 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.688741 25616 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:31.617061 26913 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.689216 26948 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.646041 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:31.716809 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70812us
I20240426 13:19:31.704427 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.676090 26938 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.718034 24954 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:31.705950 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.718871 26946 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.719396 26946 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:31.636904 26865 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20240426 13:19:31.688818 26413 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:31.709844 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.707825 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.641862 26867 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20240426 13:19:31.619905 26911 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20240426 13:19:31.715031 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.718004 26938 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.703336 26848 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:31.729727 25619 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.720227 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.730252 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.704025 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.705617 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93881.4us
I20240426 13:19:31.731400 24951 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.731956 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.718720 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.721992 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.703135 26930 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.723385 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.708740 24949 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:31.703640 26707 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:31.726478 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.737952 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.738256 24954 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.739180 24921 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.727734 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.727839 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.710888 26952 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.716980 26935 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:31.739584 25102 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:31.748082 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.740859 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.750378 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.751065 24954 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:31.751243 25102 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.743300 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.748828 25103 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:31.753012 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.753803 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.755064 24955 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:31.752781 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1255) took 1120 ms. Trace:
I20240426 13:19:31.756340 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.756583 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.756794 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.757215 24954 raft_consensus.cc:2354] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:31.757755 24951 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.756381 25101 rpcz_store.cc:276] 0426 13:19:30.632609 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.718939 (+ 86330us) service_pool.cc:225] Handling call
0426 13:19:30.763809 (+ 44870us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:30.892373 (+128564us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:30.892403 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:30.892833 (+   430us) log.cc:842] Serialized 26 byte log entry
0426 13:19:31.272600 (+379767us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:31.272653 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:31.272733 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:31.752556 (+479823us) raft_consensus.cc:1671] finished
0426 13:19:31.752580 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:31.752720 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2628,"raft.queue_time_us":3120,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":18579,"thread_start_us":379355,"threads_started":1,"wal-append.queue_time_us":619072}
I20240426 13:19:31.764791 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.735543 26791 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:31.765740 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.766042 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.766726 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.767349 24954 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.736660 26861 raft_consensus.cc:2741] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 13:19:31.771394 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.769955 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.772219 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.741652 26707 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.772490 25103 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.741708 26912 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.764278 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.778633 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.778887 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.773941 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.776533 25099 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.777513 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.743705 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.760835 26666 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.781723 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.782863 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.729045 26684 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240426 13:19:31.792735 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.803110 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:31.762740 26931 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.810107 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.827493 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.762899 26826 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:31.835366 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80299.4us
I20240426 13:19:31.836133 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:31.836215 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.836887 25027 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:31.842514 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.843322 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:31.770179 26791 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.774928 26748 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:31.741672 26930 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.835731 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.833465 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:31.842209 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.757764 26839 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20240426 13:19:31.849722 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.742233 26950 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.098s
I20240426 13:19:31.787039 26773 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.787101 26656 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.894455 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:31.896878 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60841.7us
I20240426 13:19:31.897600 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.778152 26894 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.915814 24949 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:31.929134 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95804.3us
I20240426 13:19:31.781349 26949 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:31.936576 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:31.996904 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147274us
I20240426 13:19:31.997920 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.998566 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 13:19:31.999138 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62621.9us
I20240426 13:19:31.853037 26791 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:31.918536 26934 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20240426 13:19:31.927989 26689 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.835747 26826 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.001832 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 169292us
I20240426 13:19:31.928687 26679 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.835181 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:31.849269 26931 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:32.042412 25588 leader_election.cc:398] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:32.060932 25588 leader_election.cc:401] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:31.841435 26852 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.842139 26825 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.996703 26684 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.002413 26717 raft_consensus.cc:2741] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 13:19:32.092669 24953 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:32.050110 26562 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.050643 26826 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.055349 26967 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:32.051522 26653 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.917068 26848 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.080556 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78133.5us
I20240426 13:19:31.930465 26711 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.074956 26679 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:32.079988 26689 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.934757 26773 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.115828 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 123847us
W20240426 13:19:32.113582 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174152us
I20240426 13:19:32.000380 26949 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.934942 26748 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.895043 26413 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.099414 26913 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.102870 26666 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:32.140705 25623 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.008s
I20240426 13:19:32.094476 26684 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
W20240426 13:19:32.136909 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 139092us
W20240426 13:19:32.164563 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 122378us
I20240426 13:19:32.177058 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:32.177651 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.937687 26946 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.100339 26912 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.134790 26951 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:32.111743 26562 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.138872 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.113363 26964 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.001817 26935 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.189941 24950 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:32.107462 26967 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.226334 26967 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.227591 26967 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:32.228307 26944 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:19:32.206008 26562 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.230580 26562 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.231232 26562 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.191095 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.139729 26689 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:32.208920 26957 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20240426 13:19:32.211964 25098 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.000s
I20240426 13:19:32.253829 25098 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:32.214083 26858 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.818s	user 0.007s	sys 0.002s
I20240426 13:19:32.216156 26948 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.216601 26717 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:32.217304 26872 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.626s	user 0.015s	sys 0.000s
W20240426 13:19:32.221777 26959 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:19:32.222406 26968 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:32.196799 26931 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.224573 26852 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.225080 26748 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.235486 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44449.9us
I20240426 13:19:32.253294 26681 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:32.256244 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:32.259583 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:32.206617 26684 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.271014 25624 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240426 13:19:32.231775 26562 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.273685 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:32.277537 25623 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.239830 26949 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.285768 26684 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.287578 26978 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.277686 25624 pending_rounds.cc:76] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 5
I20240426 13:19:32.288214 25624 pending_rounds.cc:97] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240426 13:19:32.288611 25624 raft_consensus.cc:2866] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:32.288161 26978 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.290578 26978 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.291515 26938 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.000s
I20240426 13:19:32.291121 26912 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.298657 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135647us
I20240426 13:19:32.271569 26858 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.237869 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:32.282743 26930 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.280411 26852 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.303175 26681 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.325064 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.270035 26717 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.284178 26711 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:32.323926 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.028s	sys 0.025s
I20240426 13:19:32.334282 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:32.343499 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.029s	sys 0.025s
W20240426 13:19:32.326954 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.000s	sys 0.006s
W20240426 13:19:32.344120 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 561365us
W20240426 13:19:32.344393 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.001s	sys 0.006s
I20240426 13:19:32.344798 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.233688 26679 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.347503 25623 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:32.330940 26941 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.812s	user 0.003s	sys 0.000s
I20240426 13:19:32.291932 26938 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.353462 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 175900us
I20240426 13:19:32.354768 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.358769 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.359226 24921 leader_election.cc:401] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:32.310592 26825 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240426 13:19:32.334071 26681 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.367245 25031 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.006s	sys 0.009s
I20240426 13:19:32.363463 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:32.344920 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 572772us
I20240426 13:19:32.371989 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:32.376855 25031 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.006s	sys 0.009s
I20240426 13:19:32.353456 26711 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.384384 25620 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:32.385783 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.387112 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.387594 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:32.388259 25623 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.387873 25096 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:32.362038 26859 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.376379 26681 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.391319 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.364526 26984 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:32.394110 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1258) took 1686 ms. Trace:
I20240426 13:19:32.406558 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.407238 25101 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.366494 26965 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:32.398500 26857 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
W20240426 13:19:32.409498 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46118us
I20240426 13:19:32.346729 26852 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.418552 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.417017 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.321807 26666 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.378826 26949 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.342223 26689 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.432618 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.426288 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:32.433701 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51217.2us
I20240426 13:19:32.427109 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.434476 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.405448 25105 rpcz_store.cc:276] 0426 13:19:30.707419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.811873 (+104454us) service_pool.cc:225] Handling call
0426 13:19:30.828545 (+ 16672us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:30.880284 (+ 51739us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:30.880335 (+    51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:30.899956 (+ 19621us) log.cc:842] Serialized 26 byte log entry
0426 13:19:31.162826 (+262870us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:31.162870 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:31.162949 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:32.292712 (+1129763us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:32.393846 (+101134us) raft_consensus.cc:1671] finished
0426 13:19:32.393867 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.394060 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27312,"raft.queue_time_us":61478,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":4303616,"thread_start_us":234549,"threads_started":1,"wal-append.queue_time_us":534891}
I20240426 13:19:32.423740 26984 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.435451 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.437206 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.423785 26857 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:32.442831 26982 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:32.443442 26982 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:32.443363 26857 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.444532 26963 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20240426 13:19:32.446514 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:32.447574 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.446421 26979 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.453267 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:32.389786 26711 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.436702 26984 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.437590 26965 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.440164 26978 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.461323 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.465350 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.467541 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.467703 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.467322 26562 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.470634 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.471233 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.472788 26789 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:32.473760 25616 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:32.463913 26748 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.475593 26987 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.475844 26938 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.476181 26987 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.471515 25098 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:32.448630 26974 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.451892 26949 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:32.457383 26979 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.490468 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.548238 25026 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:19:32.468669 26984 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.490324 26684 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.489979 26974 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.505182 26979 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.552403 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.480294 26912 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:32.502197 26949 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.451608 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:32.560742 24956 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.561108 24956 raft_consensus.cc:603] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:32.559259 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.554895 26962 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.556293 26684 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.558924 26974 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.564103 26949 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:32.577741 26994 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:32.571481 26707 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.004s
I20240426 13:19:32.563675 26413 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.444533 26982 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.570681 26717 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.636121 26987 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.623158 26684 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:32.633411 26989 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.167s	user 0.000s	sys 0.005s
I20240426 13:19:32.650985 26681 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.651804 25616 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:32.652318 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:32.657043 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:32.641599 26791 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.001s
I20240426 13:19:32.633512 26413 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.661707 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89697.1us
I20240426 13:19:32.622073 26912 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20240426 13:19:32.665623 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:32.666152 25028 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:32.696898 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77488.6us
I20240426 13:19:32.721508 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.722568 25025 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:32.646641 26852 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.586622 26962 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.640180 26707 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.669389 26413 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:32.734666 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 244) took 1010 ms. Trace:
I20240426 13:19:32.670620 26717 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:32.575757 26965 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.746850 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78184.6us
I20240426 13:19:32.748843 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:32.749063 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.725708 26711 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.757297 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:32.760192 26757 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.756309 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.762538 25028 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:32.765573 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:32.716069 26791 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.777848 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1361) took 1069 ms. Trace:
I20240426 13:19:32.778293 24954 rpcz_store.cc:276] 0426 13:19:31.708479 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.837710 (+129231us) service_pool.cc:225] Handling call
0426 13:19:32.478145 (+640435us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.478302 (+   157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:32.478336 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.478375 (+    39us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:32.478407 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.478458 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.777672 (+299214us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b54003c46d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:32.777786 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1472118016}
I20240426 13:19:32.781419 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 241) took 1058 ms. Trace:
I20240426 13:19:32.781847 24951 rpcz_store.cc:276] 0426 13:19:31.723458 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.070844 (+347386us) service_pool.cc:225] Handling call
0426 13:19:32.095465 (+ 24621us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.167155 (+ 71690us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:32.167201 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.167260 (+    59us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:32.167303 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.781173 (+613870us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b5400078fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:32.781221 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.781363 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164268416}
I20240426 13:19:32.793637 26791 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.796628 27005 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20240426 13:19:32.799319 25022 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:32.801977 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.751834 26748 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.747134 26993 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:32.807500 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58753.7us
I20240426 13:19:32.810534 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:32.811277 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62311.9us
I20240426 13:19:32.812839 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.817310 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.833312 24950 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:32.798568 26987 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.747229 26978 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20240426 13:19:32.807360 26935 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:32.877789 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65035.6us
I20240426 13:19:32.878001 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:32.873888 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.872246 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.924527 24948 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:32.921037 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 160389us
I20240426 13:19:32.884166 26967 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.885835 26984 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.900899 25096 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:32.902247 25098 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:32.816939 26993 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.906903 26707 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:32.908630 27001 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20240426 13:19:32.760720 26757 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.760937 26717 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.764483 26852 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:32.926102 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.926375 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:32.969777 26994 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.037034 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.037110 26994 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.877038 26980 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:33.038832 26980 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.040103 26980 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.044143 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:32.940512 25101 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:32.880206 26987 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.940608 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.046288 26987 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:33.046885 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106355us
I20240426 13:19:32.882421 26948 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.881093 26976 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:33.050670 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76911.9us
I20240426 13:19:32.819482 26748 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.957181 26952 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.003s	sys 0.002s
I20240426 13:19:32.935825 26935 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:33.079399 26995 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20240426 13:19:33.098426 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:33.063406 26681 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.104822 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 130245us
I20240426 13:19:32.971511 26967 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.090510 26948 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.110636 25096 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.113685 26967 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.115660 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.115072 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.079089 26952 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.082s	user 0.003s	sys 0.004s
I20240426 13:19:33.149784 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.153846 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 242) took 1430 ms (client timeout 1627 ms). Trace:
W20240426 13:19:33.162242 24955 rpcz_store.cc:269] 0426 13:19:31.723713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.727543 (+  3830us) service_pool.cc:225] Handling call
0426 13:19:33.153615 (+1426072us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.153784 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33842944}
I20240426 13:19:33.163829 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.100622 26931 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.175714 24952 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:33.066963 26852 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.090134 26976 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.110582 27023 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:33.180914 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144014us
I20240426 13:19:33.154882 26969 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.144775 27013 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.178627 26930 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.201449 24951 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:33.201853 24951 pending_rounds.cc:76] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:33.202102 24951 pending_rounds.cc:97] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20240426 13:19:33.187721 26959 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20240426 13:19:33.202756 24951 raft_consensus.cc:2866] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:33.198504 27028 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.198042 26969 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.179108 27015 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.270855 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.200539 27014 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.230s	user 0.000s	sys 0.000s
I20240426 13:19:33.199800 26748 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.273177 27022 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:19:33.275420 24955 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:33.110162 26948 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.186270 26962 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.310017 26962 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:33.195185 26994 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.277122 26984 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.276770 26930 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.310947 26994 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.313109 26994 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.314767 26948 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.277917 27028 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.280256 26969 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.280673 26992 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.218s
I20240426 13:19:33.281570 27004 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:33.350482 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1358) took 1644 ms. Trace:
I20240426 13:19:33.351063 24952 rpcz_store.cc:276] 0426 13:19:31.706195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.826927 (+120732us) service_pool.cc:225] Handling call
0426 13:19:33.154230 (+1327303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.181155 (+ 26925us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:33.350206 (+169051us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400387a50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.350402 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3051885824}
I20240426 13:19:33.353034 26993 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.320986 26999 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.689s	user 0.002s	sys 0.000s
I20240426 13:19:33.289105 27015 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.290350 26721 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:33.206863 27006 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.397s	user 0.000s	sys 0.001s
I20240426 13:19:33.274639 26413 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.206751 27013 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.201970 27023 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:33.384745 24955 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:33.388986 26684 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.729s	user 0.009s	sys 0.000s
I20240426 13:19:33.404183 27013 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.391700 26976 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.403993 26684 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.354277 26757 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.295353 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87644.4us
I20240426 13:19:33.299700 27020 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:33.300855 27010 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20240426 13:19:33.182917 26931 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.415417 26684 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.741647 24949 rpcz_store.cc:276] 0426 13:19:31.724221 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.206949 (+482728us) service_pool.cc:225] Handling call
0426 13:19:32.207506 (+   557us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.236000 (+ 28494us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:32.550981 (+314981us) log.cc:842] Serialized 14 byte log entry
0426 13:19:32.734422 (+183441us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.734475 (+    53us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:32.734503 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.734549 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.734626 (+    77us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84794,"prepare.queue_time_us":403679,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1891,"raft.queue_time_us":160551,"thread_start_us":375529,"threads_started":3}
I20240426 13:19:33.205385 27016 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.323191 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 502) took 1609 ms. Trace:
I20240426 13:19:33.417984 26931 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.332537 27024 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20240426 13:19:33.333884 27028 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.333268 26902 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:33.335453 26930 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.340250 27021 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20240426 13:19:33.398967 26979 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.834s	user 0.009s	sys 0.000s
I20240426 13:19:33.408357 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.389178 27017 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20240426 13:19:33.431195 24955 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.410717 26413 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.388716 26983 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.354785 26717 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.389729 26852 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.390372 26935 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:33.395608 26974 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.765s	user 0.008s	sys 0.000s
I20240426 13:19:33.413532 27023 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.413678 26987 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.414848 26711 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.416113 26656 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:33.417495 25620 rpcz_store.cc:276] 0426 13:19:31.713727 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.713967 (+   240us) service_pool.cc:225] Handling call
0426 13:19:31.727236 (+ 13269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.384820 (+657584us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:33.322887 (+938067us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:33.323095 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1499101952}
W20240426 13:19:33.322382 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1647) took 1753 ms (client timeout 2131 ms). Trace:
W20240426 13:19:33.288193 26826 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.170s	user 0.025s	sys 0.000s
I20240426 13:19:33.421267 26902 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:33.429812 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.419600 26721 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:33.396494 26949 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.762s	user 0.010s	sys 0.000s
I20240426 13:19:33.441612 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.434276 26931 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:33.476912 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47191.6us
I20240426 13:19:33.466962 24955 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:33.435839 27016 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.512337 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.474530 25624 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:33.513191 25022 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.513000 27016 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:33.510012 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 395144us
I20240426 13:19:33.511834 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.497443 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.497952 26893 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240426 13:19:33.504916 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.501528 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.497982 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.507184 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.497051 25028 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:33.507427 26825 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:33.519534 24955 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:33.511440 25031 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.511929 26721 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.446130 24953 rpcz_store.cc:269] 0426 13:19:31.568863 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.568979 (+   116us) service_pool.cc:225] Handling call
0426 13:19:33.322114 (+1753135us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.322297 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1075071232}
I20240426 13:19:33.510955 26747 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.524356 26747 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.469985 26656 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.433167 26979 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.522567 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.516571 26945 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:33.520395 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:33.526347 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.443212 26983 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.528553 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.529773 27016 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.531064 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.536602 27035 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.553850 24949 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:33.475145 26711 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.482452 26912 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.564662 27016 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:33.565459 26912 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.567255 26717 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:33.496501 26791 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.443020 26974 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.568310 26791 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:33.569542 26974 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.547695 24953 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:33.569240 26656 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.518529 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.549201 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.570575 26656 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.518867 26931 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:33.551681 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.552778 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.497020 26949 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.536993 25022 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:33.556130 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:33.483479 26826 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.564239 26852 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.566030 27016 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:33.550933 25029 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:33.517349 26992 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:33.547915 26902 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.701654 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:33.763484 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.697258 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:33.572139 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.702180 25101 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:33.576831 25616 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:33.633261 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.763520 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.573945 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:33.696951 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.767571 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 215963us
I20240426 13:19:33.769953 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:33.765831 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.768944 25100 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:33.770813 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.768579 25097 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
W20240426 13:19:33.772177 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 256907us
I20240426 13:19:33.773167 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.774655 25030 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:33.703209 24955 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:33.713902 24948 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:33.718569 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 206812us
I20240426 13:19:33.544389 26945 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:33.779426 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.780755 24950 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.780233 26713 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:33.789320 27013 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.790050 26902 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.789847 27013 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.790665 27029 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.791772 24921 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.792493 24921 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:33.792763 27036 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.794199 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.795046 26982 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:33.767443 26935 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.795758 26935 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.796624 26987 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.798583 26711 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.799947 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:33.702877 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.545958 27034 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.259s
I20240426 13:19:33.801546 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.804061 26747 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.546239 26721 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.658771 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:33.814680 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44812.5us
I20240426 13:19:33.815686 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.719955 27005 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20240426 13:19:33.815769 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.826670 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.827581 25030 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.570138 26791 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:33.820183 25105 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:33.772967 26969 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.817199 26681 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.828832 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.829517 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.699482 26983 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.832753 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.714738 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.830109 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.808916 26945 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.781651 27023 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.791847 27013 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.776221 27016 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.838551 25103 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:33.814940 26684 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.642743 26912 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.835191 25105 raft_consensus.cc:603] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:33.832010 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:33.844260 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.823452 26721 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.817659 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.881398 25024 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:33.860384 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.868366 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.882761 25022 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.867568 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.886647 25096 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:33.830569 26980 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.053s	sys 0.079s
I20240426 13:19:33.828869 26747 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:33.854876 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:33.887543 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55876.9us
I20240426 13:19:33.888249 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.837137 26713 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:33.891892 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.791178 27029 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.896806 25617 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:33.897078 26713 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:33.905532 26713 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:33.890435 26980 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.057s	sys 0.080s
I20240426 13:19:33.905663 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.906388 25617 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:33.906965 26980 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.907440 26980 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.842401 26979 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.890673 27023 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.886649 27031 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.909409 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.909034 27031 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.915092 27016 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.846850 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.896394 26852 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.886425 26839 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.925287 26852 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:33.927584 26988 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.464s	user 0.005s	sys 0.000s
I20240426 13:19:33.835861 27034 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20240426 13:19:33.889864 27050 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20240426 13:19:33.943784 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:33.865060 26949 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.907562 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.839051 26413 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240426 13:19:33.846675 26969 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.861094 27015 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.202142 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:34.204634 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:33.893813 26987 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:34.072948 25103 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.212214 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.948988 25026 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.207999 25030 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:34.128731 26799 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:19:34.221287 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:34.203199 26826 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.224678 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:34.226727 26982 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:34.186215 27029 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.227591 26982 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.227947 26982 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.228286 26982 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:34.226773 27044 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.334s	user 0.000s	sys 0.000s
I20240426 13:19:34.224766 27015 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:34.233008 27015 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:33.992442 26717 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.235487 25622 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.153849 26979 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.051510 26679 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:33.948782 27035 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.169818 26993 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.237046 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.177959 26983 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.239933 26949 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:34.073419 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.177299 26711 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.244072 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.889269 26684 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.272787 25623 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.271129 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.276804 24950 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.948601 26825 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.288s
I20240426 13:19:33.980809 26977 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.259s
W20240426 13:19:34.278038 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1371) took 1468 ms (client timeout 1953 ms). Trace:
W20240426 13:19:34.299245 24954 rpcz_store.cc:269] 0426 13:19:32.810106 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.810241 (+   135us) service_pool.cc:225] Handling call
0426 13:19:34.247771 (+1437530us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b5400367d28. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda96290034 00007fda9525a0a7 00007fda95259838 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:34.277984 (+ 30213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17551232}
I20240426 13:19:34.050127 26681 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.190s
I20240426 13:19:34.279827 27036 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.239567 27013 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:34.226130 26826 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240426 13:19:34.245764 26980 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.295292 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.309245 26980 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.309430 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.309732 26980 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.310393 26413 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:34.310984 26413 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:34.311420 26413 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:34.311736 26980 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.900455 26931 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.298195 26975 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:34.312691 26975 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.312359 26413 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.313086 26975 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:34.298105 26711 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.315686 26980 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.316437 26711 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.277066 26950 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.317272 26950 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.281038 26852 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.306602 25027 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.209383 26987 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.298594 26721 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.306980 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.295934 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.323061 25026 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:34.316227 26980 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.306607 25024 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:34.245963 26717 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.268276 26979 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.286134 26949 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:34.324609 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197915us
I20240426 13:19:34.273576 26679 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.333865 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.278291 27035 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.281879 26993 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.333006 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.298938 25616 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:34.307807 27013 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:34.341785 25623 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:34.318802 26711 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.319229 27031 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.360366 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.366950 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.324880 26987 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.363988 25022 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 5
I20240426 13:19:34.299352 26983 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.370616 26993 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.374055 27035 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.375001 26679 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.375994 26950 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.375640 26711 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.376346 27051 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:34.377022 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.377205 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.521199 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.521519 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.367286 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.522274 25102 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:34.522439 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:34.522531 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 155318us
I20240426 13:19:34.523149 25098 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:34.388717 26902 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.516904 26980 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.517534 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.385861 26975 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.550269 26902 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.390717 27051 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.520279 26825 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.552233 27051 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.552372 26825 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.554502 26825 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.552577 26902 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.521044 26717 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.523058 26983 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.524005 26993 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.556329 26993 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.558223 26993 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.524745 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.381350 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:34.559120 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 177839us
W20240426 13:19:34.560534 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.953s (SENT) [suppressed 3 similar messages]
I20240426 13:19:34.527467 26977 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.561517 25028 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.561741 26977 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.562474 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.563525 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.528120 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.522987 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.542665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.544155 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.527336 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.387128 26952 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.394539 26949 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.550405 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.560465 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:34.569149 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44474.1us
I20240426 13:19:34.569262 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.545830 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.569276 25104 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:34.585655 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.563970 26977 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.611073 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.609678 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.615527 25031 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:34.587918 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:34.564955 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:34.627214 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.624315 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.589632 26975 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.588342 26993 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.636812 26993 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.637255 26993 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:34.639720 26976 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:34.588188 27058 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:34.640251 26976 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:34.606973 26825 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:34.641256 26976 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.564582 26684 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.560469 26969 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.628501 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:34.626628 26936 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.632570 26977 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:34.633800 26711 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.653641 25023 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:34.655005 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:34.755527 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131274us
W20240426 13:19:34.755669 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100809us
I20240426 13:19:34.656080 25026 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:34.756855 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.748525 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.748714 26949 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.764104 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:34.766184 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:34.770835 25097 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:34.773769 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.774236 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.774986 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.776316 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:34.779820 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 219312us
W20240426 13:19:34.757576 26713 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.851s	user 0.010s	sys 0.000s
I20240426 13:19:34.784698 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.787647 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.748008 27058 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.796753 27065 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.566102 27013 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.748024 27066 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.763437 26976 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.626302 26982 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.804728 26824 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:34.779332 26980 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:34.768965 26711 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:34.816562 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:34.791646 26717 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:34.826982 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.757166 26987 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.797291 27035 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:34.780081 27067 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:34.837075 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.000s	sys 0.070s
W20240426 13:19:34.840375 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.000s	sys 0.070s
I20240426 13:19:34.819867 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:34.854087 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.854305 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.809082 27072 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:34.858664 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.855458 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.862185 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.863806 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.864714 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.799546 26713 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.867205 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.820407 26980 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.868291 25623 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:34.854871 26931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.006s	sys 0.019s
I20240426 13:19:34.877615 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:34.835172 26824 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.879892 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.842530 27066 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.900462 25101 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.890446 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.848479 26711 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.856490 27036 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.853963 26979 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.801903 27058 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.878357 27065 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.878167 26980 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:34.877869 27072 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.877125 26717 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:34.888494 26931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.006s	sys 0.019s
I20240426 13:19:34.927772 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 18) took 2185 ms. Trace:
I20240426 13:19:34.870842 27068 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:34.607136 26952 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:34.906419 26799 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 2.460s	user 0.028s	sys 0.029s
I20240426 13:19:34.898041 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.753995 26936 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.943436 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.937728 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:34.938556 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:34.940542 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.029767 24950 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.863616 27067 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20240426 13:19:34.940193 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:34.880633 26849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.009s	sys 0.022s
I20240426 13:19:34.906159 27066 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.935213 26902 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:35.006626 26824 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.008947 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.047s	sys 0.051s
W20240426 13:19:35.050403 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.048s	sys 0.051s
W20240426 13:19:35.013350 27070 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20240426 13:19:35.016868 26945 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.156s	user 0.008s	sys 0.003s
I20240426 13:19:35.018179 26717 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240426 13:19:35.022079 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 219415us
I20240426 13:19:35.020874 25105 rpcz_store.cc:276] 0426 13:19:32.750571 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.750701 (+   130us) service_pool.cc:225] Handling call
0426 13:19:34.921932 (+2171231us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:34.922065 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2443024256}
I20240426 13:19:35.028427 26773 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.031198 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 323) took 1554 ms. Trace:
I20240426 13:19:34.940542 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:35.031569 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53497.7us
I20240426 13:19:35.039399 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.031858 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:35.041694 24949 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
W20240426 13:19:35.050745 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 848245us
W20240426 13:19:35.051062 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.022s	sys 0.040s
I20240426 13:19:35.056715 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.089732 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.023s	sys 0.040s
I20240426 13:19:35.090270 24956 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:35.023082 25026 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:35.001046 27065 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.060214 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:35.043989 26849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.009s	sys 0.022s
I20240426 13:19:35.066220 25620 rpcz_store.cc:276] 0426 13:19:33.476545 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.476687 (+   142us) service_pool.cc:225] Handling call
0426 13:19:34.943255 (+1466568us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:35.031118 (+ 87863us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12377984}
I20240426 13:19:35.086238 24950 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:35.091915 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:35.114271 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:35.099046 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.103884 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43762.7us
I20240426 13:19:35.052759 26902 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.117491 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.046649 25616 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.964613 27036 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.105474 26839 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.106870 26799 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:35.107797 26893 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:35.102306 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:35.110415 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 241193us
I20240426 13:19:35.103863 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.057150 27067 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.115105 26982 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.062566 26945 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.127070 27070 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:19:35.118350 26858 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20240426 13:19:35.121434 26983 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:35.160833 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.161780 26980 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.162204 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.119546 26773 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.164192 26979 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.164410 26773 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.164670 26979 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.166327 26773 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.129403 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.166937 26969 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.167454 26969 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.162992 25102 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:35.169287 26969 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.061219 27035 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20240426 13:19:35.169960 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.170272 25624 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:35.170814 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 326) took 1657 ms. Trace:
I20240426 13:19:35.170877 27058 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.171088 27035 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:35.171525 27035 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.150868 24956 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:35.167670 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.110191 27065 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:35.122491 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.172358 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 228) took 2794 ms (client timeout 1954 ms). Trace:
I20240426 13:19:35.130292 27066 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.151640 24996 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:35.248315 24996 leader_election.cc:401] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20240426 13:19:35.249303 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97848.9us
I20240426 13:19:35.250299 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.154198 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1296) took 1173 ms. Trace:
I20240426 13:19:35.120647 26902 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:35.128674 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.251188 25022 rpcz_store.cc:276] 0426 13:19:33.984277 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.984499 (+   222us) service_pool.cc:225] Handling call
0426 13:19:35.154006 (+1169507us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.154134 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347513728}
W20240426 13:19:35.152184 27077 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20240426 13:19:35.199407 26993 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.006s
I20240426 13:19:35.198844 27035 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:35.178643 25030 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:35.148802 26936 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.158808 26945 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.157219 26982 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.179050 26839 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.171644 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.180002 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:35.123345 26711 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.144268 26987 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.173192 26825 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.149931 26799 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:35.122749 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 330) took 1222 ms. Trace:
I20240426 13:19:35.287307 25617 rpcz_store.cc:276] 0426 13:19:33.898438 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.907008 (+  8570us) service_pool.cc:225] Handling call
0426 13:19:33.907270 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.907464 (+   194us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:34.288864 (+381400us) log.cc:842] Serialized 14 byte log entry
0426 13:19:34.815220 (+526356us) log.cc:842] Serialized 14 byte log entry
0426 13:19:35.059604 (+244384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:35.059727 (+   123us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:35.070772 (+ 11045us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.110078 (+ 39306us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.120510 (+ 10432us) inbound_call.cc:173] Queueing success response
0426 13:19:35.165401 (+ 44891us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x7b18001fdd40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":27372,"prepare.queue_time_us":418073,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":154380,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":10079,"spinlock_wait_cycles":22158080,"thread_start_us":1041913,"threads_started":3,"wal-append.queue_time_us":552014}
I20240426 13:19:35.268795 26931 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.292573 26931 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.295153 26931 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.265004 27062 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.298883 27062 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.301277 27062 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.301879 26825 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.304455 26825 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.166344 26979 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.182878 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1389) took 1731 ms. Trace:
I20240426 13:19:35.306886 24953 rpcz_store.cc:276] 0426 13:19:33.451039 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.570350 (+119311us) service_pool.cc:225] Handling call
0426 13:19:33.570606 (+   256us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.638144 (+ 67538us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.638188 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.638243 (+    55us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:33.638295 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.182593 (+1544298us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.182654 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.182813 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6837120}
I20240426 13:19:35.308751 26839 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.309432 26839 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.309962 26839 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.310863 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.152271 27067 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.184096 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1283) took 1612 ms. Trace:
I20240426 13:19:35.312388 25096 rpcz_store.cc:276] 0426 13:19:33.571872 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.573647 (+  1775us) service_pool.cc:225] Handling call
0426 13:19:35.183873 (+1610226us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.184017 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259534208}
I20240426 13:19:35.313400 27067 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.314126 25096 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:35.130584 26713 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.185982 26976 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:35.171985 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68202.6us
I20240426 13:19:35.186865 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1380) took 2371 ms. Trace:
I20240426 13:19:35.318087 24951 rpcz_store.cc:276] 0426 13:19:32.815092 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.034165 (+219073us) service_pool.cc:225] Handling call
0426 13:19:33.034473 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.288686 (+254213us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.186603 (+1897917us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.186788 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383228160}
I20240426 13:19:35.188735 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 244) took 1083 ms. Trace:
I20240426 13:19:35.319303 25025 rpcz_store.cc:276] 0426 13:19:34.105758 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.105929 (+   171us) service_pool.cc:225] Handling call
0426 13:19:35.188060 (+1082131us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:35.188255 (+   195us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.188424 (+   169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.188453 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:35.188496 (+    43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:35.188529 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.188583 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.188675 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11662080}
I20240426 13:19:35.189538 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:35.196239 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70644us
I20240426 13:19:35.321198 25028 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240426 13:19:35.193737 27030 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.671s	user 0.007s	sys 0.006s
I20240426 13:19:35.163739 26980 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.171186 25624 rpcz_store.cc:276] 0426 13:19:33.513823 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.514602 (+   779us) service_pool.cc:225] Handling call
0426 13:19:35.169909 (+1655307us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:35.170072 (+   163us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.170616 (+   544us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.170768 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5041920}
W20240426 13:19:35.122315 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 244804us
I20240426 13:19:35.329221 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:35.246212 25029 rpcz_store.cc:269] 0426 13:19:32.377809 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.377936 (+   127us) service_pool.cc:225] Handling call
0426 13:19:35.172152 (+2794216us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.172279 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2634617728}
I20240426 13:19:35.330701 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:35.149492 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.152274 27072 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.254684 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.331712 24949 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:35.255227 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.256639 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.332902 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.333544 24950 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:35.333777 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.334340 24956 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:35.129945 27036 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.265480 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.331257 25029 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:35.268913 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 325) took 1756 ms. Trace:
I20240426 13:19:35.274653 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.277336 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.279120 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:35.257344 27038 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:35.259568 26912 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:35.277093 26977 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240426 13:19:35.268287 27061 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.283840 26945 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:35.284787 26679 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:35.286015 26799 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:35.308674 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.314896 26976 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.199510 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 333221us
I20240426 13:19:35.325906 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.326475 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.377606 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.362643 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:35.368499 27086 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.375301 24949 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:35.261610 26993 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.371312 26975 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.336354 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:35.362401 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 107793us
I20240426 13:19:35.368892 26979 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:35.375041 26978 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:35.336827 25619 rpcz_store.cc:276] 0426 13:19:33.512289 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.512689 (+   400us) service_pool.cc:225] Handling call
0426 13:19:35.179851 (+1667162us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:35.268867 (+ 89016us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9327488}
I20240426 13:19:35.323515 27066 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.394171 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.368366 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:35.376256 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:35.279924 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.028748 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.277731 26967 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.007s	user 0.021s	sys 0.018s
I20240426 13:19:35.275308 26982 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.030153 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.061208 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1294) took 2134 ms. Trace:
I20240426 13:19:36.031062 25624 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:35.666205 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 295231us
I20240426 13:19:36.061609 25027 rpcz_store.cc:276] 0426 13:19:33.897916 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.898044 (+   128us) service_pool.cc:225] Handling call
0426 13:19:33.898401 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.318923 (+420522us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.031844 (+1712921us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000b0c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:36.031992 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938320768}
I20240426 13:19:36.031661 25029 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.029567 24949 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:35.528196 27039 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.063685 24949 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.063580 27039 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.031929 25103 raft_consensus.cc:595] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.066128 27039 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.437134 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 252) took 2621 ms. Trace:
I20240426 13:19:36.067956 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1303) took 1242 ms. Trace:
I20240426 13:19:36.066859 24947 rpcz_store.cc:276] 0426 13:19:32.815520 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.035890 (+220370us) service_pool.cc:225] Handling call
0426 13:19:33.054403 (+ 18513us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.078236 (+ 23833us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:33.509856 (+431620us) log.cc:842] Serialized 14 byte log entry
0426 13:19:33.768914 (+259058us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.768959 (+    45us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:33.768994 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.423670 (+1654676us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.423757 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.437063 (+ 13306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148281,"prepare.queue_time_us":112979,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":131218,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"spinlock_wait_cycles":55118976,"thread_start_us":524430,"threads_started":2,"wal-append.queue_time_us":319100,"wal-append.run_cpu_time_us":1874,"wal-append.run_wall_time_us":1190160}
I20240426 13:19:36.068450 25103 rpcz_store.cc:276] 0426 13:19:34.825636 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.067057 (+1241421us) service_pool.cc:225] Handling call
0426 13:19:36.067384 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.067572 (+   188us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.067607 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.067660 (+    53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.067712 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.067775 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.067895 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.033568 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:36.025820 25620 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.008s	sys 0.032s
W20240426 13:19:36.070310 25620 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.008s	sys 0.032s
I20240426 13:19:36.034663 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1297) took 1816 ms. Trace:
I20240426 13:19:36.071293 25030 rpcz_store.cc:276] 0426 13:19:34.218200 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.219330 (+  1130us) service_pool.cc:225] Handling call
0426 13:19:35.172949 (+953619us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.259894 (+ 86945us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.034424 (+774530us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54001059d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:36.034587 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2192438528}
I20240426 13:19:35.361764 27036 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.074041 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1449) took 1457 ms. Trace:
I20240426 13:19:36.074291 25023 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:36.056422 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 718569us
I20240426 13:19:35.669252 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.151542 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 308) took 1544 ms. Trace:
I20240426 13:19:36.144665 25097 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:36.062155 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 311) took 1438 ms. Trace:
I20240426 13:19:36.152168 25098 rpcz_store.cc:276] 0426 13:19:34.616943 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.062789 (+1445846us) service_pool.cc:225] Handling call
0426 13:19:36.073560 (+ 10771us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.073742 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.073768 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.073805 (+    37us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.073838 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.073904 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.073999 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.153529 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 310) took 1537 ms. Trace:
I20240426 13:19:36.153407 25102 rpcz_store.cc:276] 0426 13:19:34.623705 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061469 (+1437764us) service_pool.cc:225] Handling call
0426 13:19:36.061728 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.061875 (+   147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.061898 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.061935 (+    37us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.061967 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.062016 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.062108 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:35.807083 26711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.003s	sys 0.028s
I20240426 13:19:36.156773 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.377200 26965 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.030241 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1757) took 1137 ms. Trace:
W20240426 13:19:35.855255 27051 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.247s	user 0.009s	sys 0.001s
I20240426 13:19:36.153179 25105 rpcz_store.cc:276] 0426 13:19:34.607060 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061414 (+1454354us) service_pool.cc:225] Handling call
0426 13:19:36.150997 (+ 89583us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.151201 (+   204us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.151225 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.151263 (+    38us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.151300 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.151363 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.151487 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.010030 27085 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.157493 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:36.187183 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.187813 25623 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:36.028354 26987 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:36.188236 26982 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:36.188839 26902 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.930s	user 0.010s	sys 0.000s
I20240426 13:19:36.188707 26987 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.189239 26902 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.190052 26993 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.190619 26987 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.191051 26902 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.191164 26987 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:36.157095 26711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.005s	sys 0.034s
I20240426 13:19:36.184367 26965 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.194061 27035 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.927s	user 0.009s	sys 0.000s
I20240426 13:19:36.194439 27035 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.181253 26979 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:36.195214 26965 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.060376 26983 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:36.197932 26987 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:36.169730 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107400us
I20240426 13:19:36.179512 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.226186 24956 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:36.171295 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 137797us
I20240426 13:19:36.075244 26824 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:35.865967 26839 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.159194 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:36.179126 25026 rpcz_store.cc:276] 0426 13:19:34.892546 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.125095 (+232549us) service_pool.cc:225] Handling call
0426 13:19:35.127254 (+  2159us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:35.154368 (+ 27114us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.154399 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:35.154804 (+   405us) log.cc:842] Serialized 26 byte log entry
0426 13:19:35.275881 (+121077us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:35.275925 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.275990 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:36.030056 (+754066us) raft_consensus.cc:1671] finished
0426 13:19:36.030076 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.030191 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22080,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":120721,"threads_started":1,"wal-append.queue_time_us":179749}
W20240426 13:19:35.419800 26949 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.000s
I20240426 13:19:36.158506 25029 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.185180 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1689) took 1087 ms. Trace:
I20240426 13:19:36.233538 24949 rpcz_store.cc:276] 0426 13:19:35.097293 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.072362 (+975069us) service_pool.cc:225] Handling call
0426 13:19:36.184741 (+112379us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.184894 (+   153us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.184918 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.184959 (+    41us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.184988 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.185033 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.185125 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.197865 26983 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.198321 24953 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:19:35.418217 27091 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.424367 27088 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.154026 25097 rpcz_store.cc:276] 0426 13:19:34.615889 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061301 (+1445412us) service_pool.cc:225] Handling call
0426 13:19:36.061587 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.153317 (+ 91730us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.153483 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:35.848652 27089 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:36.055588 26967 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.850s	user 0.025s	sys 0.037s
W20240426 13:19:36.199817 27065 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.942s	user 0.006s	sys 0.002s
W20240426 13:19:36.211091 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.009s	sys 0.011s
I20240426 13:19:36.073561 27038 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.061029 27066 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.168584 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.215368 25618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.008s	sys 0.015s
W20240426 13:19:36.215528 26865 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.003s	sys 0.021s
I20240426 13:19:36.168956 27036 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.059867 27061 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.230868 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:36.235603 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1697) took 1126 ms. Trace:
W20240426 13:19:36.251989 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.009s	sys 0.011s
W20240426 13:19:36.300851 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.02181e+06us
W20240426 13:19:36.303504 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72754.8us
I20240426 13:19:36.304239 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:36.305879 24953 rpcz_store.cc:276] 0426 13:19:35.109480 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.197787 (+1088307us) service_pool.cc:225] Handling call
0426 13:19:36.198110 (+   323us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.235306 (+ 37196us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.235543 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.060484 26975 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.201488 27051 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.059976 27086 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.228746 27035 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.231669 26824 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.204097 27085 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.241989 26839 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:36.309499 24951 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:36.309828 24995 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.310035 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1703) took 1005 ms. Trace:
I20240426 13:19:36.547811 24949 rpcz_store.cc:276] 0426 13:19:35.304953 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.253172 (+948219us) service_pool.cc:225] Handling call
0426 13:19:36.309420 (+ 56248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.309661 (+   241us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.309689 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.309742 (+    53us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.309777 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.309845 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.309976 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1937,"raft.queue_time_us":109,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":14558}
W20240426 13:19:36.253688 25618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.008s	sys 0.015s
I20240426 13:19:36.328989 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.316252 26902 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:36.295958 26865 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.009s	sys 0.026s
I20240426 13:19:36.414971 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.416005 27051 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.417570 27086 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.418565 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.418862 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.014s	sys 0.085s
W20240426 13:19:36.585656 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.014s	sys 0.085s
I20240426 13:19:36.413208 26824 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:36.420321 27099 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
W20240426 13:19:36.421087 27100 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:19:36.423743 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.424587 27104 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20240426 13:19:36.425393 26983 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:36.427173 25621 raft_consensus.cc:2354] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:36.272291 27089 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.253994 27091 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.326050 26993 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:36.271749 27065 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.302289 26965 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.133s
I20240426 13:19:36.432430 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.253886 27088 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.440493 27013 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.440912 27066 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.478425 24948 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:36.547690 24995 leader_election.cc:401] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:36.630605 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 320895us
I20240426 13:19:36.631376 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.295903 27038 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.661561 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.014s	sys 0.015s
I20240426 13:19:36.412925 26977 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.595196 24951 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:36.609331 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 279) took 1284 ms. Trace:
I20240426 13:19:36.609603 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.547623 26976 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.574102 27031 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.667058 24949 rpcz_store.cc:276] 0426 13:19:35.324559 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.574410 (+1249851us) service_pool.cc:225] Handling call
0426 13:19:36.608788 (+ 34378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.608962 (+   174us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.608996 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.609042 (+    46us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.609099 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.609157 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.609273 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:36.616858 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.010s	sys 0.011s
W20240426 13:19:36.620208 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 590136us
W20240426 13:19:36.627364 25620 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.034s	sys 0.017s
W20240426 13:19:36.680219 25620 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.034s	sys 0.017s
I20240426 13:19:36.613315 26861 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:36.629233 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1289) took 2855 ms. Trace:
I20240426 13:19:36.663522 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.681918 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.682835 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.683045 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:36.419577 26978 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.684731 26982 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:36.684803 26978 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.685382 26982 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:36.686245 26982 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.686914 26978 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.687436 26978 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:36.683895 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.688766 26839 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:36.667596 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46627.1us
I20240426 13:19:36.681195 25100 rpcz_store.cc:276] 0426 13:19:33.765205 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.771339 (+  6134us) service_pool.cc:225] Handling call
0426 13:19:33.836819 (+ 65480us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.844223 (+  7404us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.844250 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.844293 (+    43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:33.844333 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.620740 (+2776407us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b54003ef8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:36.620779 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.620897 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":896,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":109,"spinlock_wait_cycles":13819264}
I20240426 13:19:36.691263 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:36.692090 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1314) took 2133 ms (client timeout 2137 ms). Trace:
W20240426 13:19:36.692394 25031 rpcz_store.cc:269] 0426 13:19:34.559039 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.629004 (+ 69965us) service_pool.cc:225] Handling call
0426 13:19:36.691120 (+2062116us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:36.692020 (+   900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16325504}
I20240426 13:19:36.695515 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1410) took 1365 ms. Trace:
I20240426 13:19:36.698159 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1701) took 1428 ms. Trace:
I20240426 13:19:36.621492 26993 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:36.698505 24952 rpcz_store.cc:276] 0426 13:19:35.269721 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.310574 (+ 40853us) service_pool.cc:225] Handling call
0426 13:19:36.698011 (+1387437us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:36.698109 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1949331712}
W20240426 13:19:36.664569 27105 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20240426 13:19:36.684254 27066 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:36.695919 24954 rpcz_store.cc:276] 0426 13:19:35.329846 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.694819 (+1364973us) service_pool.cc:225] Handling call
0426 13:19:36.695094 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.695242 (+   148us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.695266 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.695304 (+    38us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.695331 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.695376 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.695467 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.619985 27061 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:36.702880 24952 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:36.720887 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.743175 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.743791 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.750771 25098 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.707871 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.662499 27086 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.704212 25023 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:36.718420 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:36.665078 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.015s	sys 0.015s
I20240426 13:19:36.757270 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.704002 26993 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.748834 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:36.749678 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47040.3us
I20240426 13:19:36.763738 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.668561 26902 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:36.752971 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.753173 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.753861 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1414) took 1360 ms. Trace:
I20240426 13:19:36.768059 24954 rpcz_store.cc:276] 0426 13:19:35.393568 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.742515 (+1348947us) service_pool.cc:225] Handling call
0426 13:19:36.753377 (+ 10862us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.753529 (+   152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.753554 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.753607 (+    53us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.753636 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.753694 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.753813 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.769028 27061 raft_consensus.cc:979] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:36.769543 27061 raft_consensus.cc:1067] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:36.769464 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.754170 25102 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:36.754894 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.772015 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:36.714111 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.773069 25027 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.745311 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1411) took 1415 ms. Trace:
I20240426 13:19:36.776222 24952 rpcz_store.cc:276] 0426 13:19:35.330234 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.702097 (+1371863us) service_pool.cc:225] Handling call
0426 13:19:36.702457 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.745035 (+ 42578us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.745253 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.742231 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 327) took 3175 ms. Trace:
I20240426 13:19:36.779903 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.779196 25105 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:36.753178 25097 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:36.772480 26935 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.777549 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.630956 26987 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20240426 13:19:36.786415 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.791716 27066 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:36.792526 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.698632 26977 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.793731 24952 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240426 13:19:36.788050 25026 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.795714 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.745251 26839 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20240426 13:19:36.796686 27066 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.797888 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:36.798816 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.800282 25026 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.805495 24952 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
I20240426 13:19:36.806625 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:36.806696 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.807363 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.810730 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1415) took 1405 ms. Trace:
I20240426 13:19:36.811187 24952 rpcz_store.cc:276] 0426 13:19:35.405266 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.809198 (+1403932us) service_pool.cc:225] Handling call
0426 13:19:36.809551 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.809758 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.809795 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.810494 (+   699us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.810527 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.810578 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.810670 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.814061 24952 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:36.814620 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1417) took 1341 ms. Trace:
I20240426 13:19:36.814983 24952 rpcz_store.cc:276] 0426 13:19:35.473312 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.812289 (+1338977us) service_pool.cc:225] Handling call
0426 13:19:36.812652 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.814363 (+  1711us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.814551 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.818254 25100 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:36.779719 27111 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:36.784384 26950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.000s	sys 0.023s
I20240426 13:19:36.837868 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.794140 26902 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.702329 27031 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.839035 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 384035us
I20240426 13:19:36.802189 26975 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:36.817011 24921 connection.cc:232] RPC call timeout handler was delayed by -1.147353270999929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
I20240426 13:19:36.799042 27113 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.851485 25100 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:36.853802 25031 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:36.870189 25031 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:36.870632 25031 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:36.805096 26949 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:36.841096 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 416246us
I20240426 13:19:36.817179 27036 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:36.862633 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 532475us
I20240426 13:19:36.865242 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.839133 27112 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.875115 25098 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:36.877003 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 192917us
I20240426 13:19:36.839814 26978 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:36.856061 26902 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:36.842197 26950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.000s	sys 0.023s
I20240426 13:19:36.884964 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.880447 25098 raft_consensus.cc:603] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:36.887799 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.888418 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.894291 25102 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.893289 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.895794 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.861876 27035 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.926415 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.909869 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.928395 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.898924 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.845811 27111 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.878069 27113 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.935175 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.044605 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1770) took 1707 ms. Trace:
I20240426 13:19:37.129886 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1432) took 2613 ms. Trace:
W20240426 13:19:37.174762 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 239703us
I20240426 13:19:36.939993 27065 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.188009 26773 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.188303 26857 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.028213 26799 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 1.978s	user 0.018s	sys 0.032s
I20240426 13:19:37.174700 25099 rpcz_store.cc:276] 0426 13:19:34.516186 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.520893 (+  4707us) service_pool.cc:225] Handling call
0426 13:19:34.521153 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.555169 (+ 34016us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:34.555197 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:34.555234 (+    37us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:34.555261 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.129580 (+2574319us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.129643 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.129812 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77845760}
I20240426 13:19:37.176631 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.978428 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.974815 25027 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.174362 25029 rpcz_store.cc:276] 0426 13:19:35.337590 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.152050 (+814460us) service_pool.cc:225] Handling call
0426 13:19:36.152323 (+   273us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.236728 (+ 84405us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.032374 (+795646us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.044536 (+ 12162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13704448}
I20240426 13:19:37.196390 25022 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:37.173280 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 308150us
I20240426 13:19:37.197623 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.198055 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.889300 27085 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.180281 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 246896us
W20240426 13:19:37.133129 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1704) took 1827 ms (client timeout 2368 ms). Trace:
W20240426 13:19:37.201211 24951 rpcz_store.cc:269] 0426 13:19:35.305273 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.327001 (+ 21728us) service_pool.cc:225] Handling call
0426 13:19:37.132984 (+1805983us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b540030c5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.133076 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2253267712}
I20240426 13:19:36.880402 26949 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.901062 27036 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:37.024406 27031 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.005756 26744 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:37.211788 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.020212 27091 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.047942 27054 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.663s	user 0.018s	sys 0.052s
I20240426 13:19:37.198654 25029 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.219026 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.219911 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:37.054740 27118 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20240426 13:19:37.054069 27124 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:19:36.896916 27089 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.204207 27023 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.204900 26949 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.203320 26656 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:37.005424 27038 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.958014 26982 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.048481 27068 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:37.179774 27016 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.213786 26713 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.208388 26952 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.194048 26824 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.580s	user 0.008s	sys 0.001s
I20240426 13:19:37.055158 26977 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.175736 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:37.253698 27120 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.208s	user 0.000s	sys 0.002s
I20240426 13:19:37.201656 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:37.262356 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 60815.1us
I20240426 13:19:37.263113 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.199110 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.208845 26945 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20240426 13:19:37.217475 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1696) took 2108 ms. Trace:
W20240426 13:19:37.197508 26799 ts_tablet_manager.cc:1430] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Tablet startup took 2163ms
I20240426 13:19:36.997896 27088 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.963431 27102 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.879437 27116 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.272281 25616 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:37.271976 27116 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.203171 27013 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.000s
I20240426 13:19:37.275219 26839 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.223158 26744 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.275925 26839 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.276865 27102 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.899456 26912 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.307s
I20240426 13:19:37.227000 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:37.279635 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52674.2us
I20240426 13:19:37.229329 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1444) took 2710 ms. Trace:
W20240426 13:19:37.231935 27119 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20240426 13:19:37.209331 26859 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20240426 13:19:37.281764 25104 rpcz_store.cc:276] 0426 13:19:34.519233 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.607133 (+ 87900us) service_pool.cc:225] Handling call
0426 13:19:34.607409 (+   276us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.864375 (+256966us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.229095 (+2364720us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.229266 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577283456}
I20240426 13:19:37.234086 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1766) took 1909 ms. Trace:
I20240426 13:19:37.283167 25025 rpcz_store.cc:276] 0426 13:19:35.324801 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.062435 (+737634us) service_pool.cc:225] Handling call
0426 13:19:36.180716 (+118281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.414263 (+233547us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.414306 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.414359 (+    53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.414398 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.233850 (+819452us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.233894 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.234020 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536591744}
I20240426 13:19:37.238317 27107 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:19:37.242096 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1412) took 1904 ms (client timeout 1953 ms). Trace:
I20240426 13:19:37.245050 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1678) took 3705 ms. Trace:
I20240426 13:19:37.226805 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.286799 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.285553 24955 rpcz_store.cc:276] 0426 13:19:33.539949 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.775670 (+235721us) service_pool.cc:225] Handling call
0426 13:19:33.775929 (+   259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:33.811202 (+ 35273us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.811245 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:33.819449 (+  8204us) log.cc:842] Serialized 26 byte log entry
0426 13:19:34.273938 (+454489us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:34.386350 (+112412us) log.cc:842] Serialized 14 byte log entry
0426 13:19:34.386550 (+   200us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:34.386650 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.047404 (+2660754us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:37.244706 (+197302us) raft_consensus.cc:1671] finished
0426 13:19:37.244842 (+   136us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.244982 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15062,"prepare.queue_time_us":24961,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":3772,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":129527,"spinlock_wait_cycles":3638016,"thread_start_us":516995,"threads_started":2,"wal-append.queue_time_us":469701}
I20240426 13:19:37.247270 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.040668 25031 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:37.289191 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91206.8us
I20240426 13:19:37.289711 25031 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:37.289911 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.289816 26902 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.259022 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.127960 27136 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.299867 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.292002 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.889837 27112 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.264549 25027 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:37.258175 27068 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:37.269091 24995 connection.cc:232] RPC call timeout handler was delayed by -1.281306637000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20240426 13:19:37.264547 24947 rpcz_store.cc:276] 0426 13:19:35.109203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.072644 (+963441us) service_pool.cc:225] Handling call
0426 13:19:36.073127 (+   483us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.679712 (+606585us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.682070 (+  2358us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:36.682536 (+   466us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.869180 (+186644us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.869236 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.869327 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.217223 (+347896us) raft_consensus.cc:1671] finished
0426 13:19:37.217249 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.217412 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114614,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":791499,"threads_started":2,"wal-append.queue_time_us":346172}
I20240426 13:19:37.270334 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.271302 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.277280 26744 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.261895 26824 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.290319 27020 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:37.291225 25099 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:37.291533 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 115871us
I20240426 13:19:37.262969 26983 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.257988 26982 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:37.284801 24948 rpcz_store.cc:269] 0426 13:19:35.337353 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.362476 (+ 25123us) service_pool.cc:225] Handling call
0426 13:19:37.241938 (+1879462us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.242032 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":490422144}
I20240426 13:19:37.223026 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.294113 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.290885 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1698) took 2181 ms. Trace:
I20240426 13:19:37.323462 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.302937 25022 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:37.303566 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:37.308207 25029 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.292995 27107 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.301532 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.280129 26747 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.339668 25025 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:36.966812 27035 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.310973 25027 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.318548 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.268733 27038 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.319650 25623 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.324744 25103 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.371992 26839 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.372124 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.297217 27082 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:37.373929 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78832.1us
I20240426 13:19:37.381572 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:37.381723 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63242.4us
I20240426 13:19:37.321910 24956 rpcz_store.cc:276] 0426 13:19:35.113229 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.226961 (+1113732us) service_pool.cc:225] Handling call
0426 13:19:36.227415 (+   454us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.233971 (+  6556us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.234014 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:36.241602 (+  7588us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.745715 (+504113us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:36.745750 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.745818 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.231365 (+485547us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:37.290626 (+ 59261us) raft_consensus.cc:1671] finished
0426 13:19:37.290643 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.290829 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24727,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":41652,"spinlock_wait_cycles":18320128,"thread_start_us":417203,"threads_started":1,"wal-append.queue_time_us":669418}
I20240426 13:19:37.387336 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.312820 26969 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:37.393986 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1295) took 3277 ms (client timeout 1884 ms). Trace:
I20240426 13:19:37.308701 27013 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.308833 27088 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.262408 26713 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.296770 27136 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:37.312767 26824 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.446843 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.388936 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.375007 27066 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.392496 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.322721 26982 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.431417 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.436323 26977 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.463347 27066 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.463909 27066 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.464471 26902 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.459601 27102 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.385877 27082 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.466312 27082 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.469377 25104 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:19:37.448412 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59578.4us
I20240426 13:19:37.373896 26747 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:37.394480 25101 rpcz_store.cc:269] 0426 13:19:34.107210 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.206033 (+ 98823us) service_pool.cc:225] Handling call
0426 13:19:37.384803 (+3178770us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.384905 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429277056}
W20240426 13:19:37.444187 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52640.1us
I20240426 13:19:37.333030 27020 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:37.477062 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 84594.1us
I20240426 13:19:37.379290 27085 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.090s
I20240426 13:19:37.474153 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.310187 27068 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.472790 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.442288 26839 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.472499 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.501579 26839 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.502158 26839 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.504021 26679 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:37.504526 26679 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.505389 26679 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.506542 26679 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:36.780496 25618 rpcz_store.cc:276] 0426 13:19:33.523234 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.525619 (+  2385us) service_pool.cc:225] Handling call
0426 13:19:33.537271 (+ 11652us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:33.553639 (+ 16368us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.556097 (+  2458us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:33.701065 (+144968us) log.cc:842] Serialized 26 byte log entry
0426 13:19:35.376276 (+1675211us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b1800118cc0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:36.585415 (+1209139us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.605122 (+ 19707us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.613581 (+  8459us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:36.616099 (+  2518us) raft_consensus.cc:1671] finished
0426 13:19:36.617100 (+  1001us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.698977 (+ 81877us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8665,"raft.queue_time_us":12524,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":105532,"spinlock_wait_cycles":65238912,"thread_start_us":1195068,"threads_started":1,"wal-append.queue_time_us":2631503}
I20240426 13:19:37.473177 27107 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.509245 27107 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.510643 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:37.511656 26839 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.512972 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.513911 27066 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.516214 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1409) took 2187 ms. Trace:
I20240426 13:19:37.516595 24949 rpcz_store.cc:276] 0426 13:19:35.328719 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.671982 (+1343263us) service_pool.cc:225] Handling call
0426 13:19:36.672433 (+   451us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.743217 (+ 70784us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.743251 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.743293 (+    42us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:36.743324 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.515956 (+772632us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.516017 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.516155 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162317568}
I20240426 13:19:37.482522 27020 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.485625 26721 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:37.529616 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:37.469175 26983 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:37.538275 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1769) took 2212 ms. Trace:
I20240426 13:19:37.458753 27035 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.489085 26987 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:37.477099 26747 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.489969 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 173) took 6736 ms. Trace:
I20240426 13:19:37.491019 27082 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.492165 25097 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.492223 26978 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.496847 24851 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.499181 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1447) took 2958 ms. Trace:
I20240426 13:19:37.468171 27096 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.514135 26839 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.485174 26713 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:37.520799 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48076.3us
W20240426 13:19:37.601756 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48046us
I20240426 13:19:37.486240 27112 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.478293 27051 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20240426 13:19:37.477502 27088 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:37.532476 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 241) took 3645 ms (client timeout 2754 ms). Trace:
W20240426 13:19:37.630228 25024 rpcz_store.cc:269] 0426 13:19:33.886835 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.887933 (+  1098us) service_pool.cc:225] Handling call
0426 13:19:37.532307 (+3644374us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54001e3554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.532409 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":947230592}
I20240426 13:19:37.540129 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 480) took 2266 ms. Trace:
I20240426 13:19:37.540645 24996 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
W20240426 13:19:37.631652 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91137.5us
I20240426 13:19:37.541411 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 278) took 2217 ms. Trace:
I20240426 13:19:37.550189 27068 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.538652 25030 rpcz_store.cc:276] 0426 13:19:35.325380 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.150920 (+825540us) service_pool.cc:225] Handling call
0426 13:19:36.151166 (+   246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.194355 (+ 43189us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.194385 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.194421 (+    36us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.194450 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.538091 (+1343641us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400056ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.538131 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.538227 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98935552}
I20240426 13:19:37.546599 27141 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:37.638420 27141 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.537609 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1762) took 2213 ms. Trace:
I20240426 13:19:37.591857 27115 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.442237 26969 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.555334 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.645700 25101 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:37.588238 25096 rpcz_store.cc:276] 0426 13:19:34.540310 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.315001 (+774691us) service_pool.cc:225] Handling call
0426 13:19:35.315292 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.332180 (+1016888us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.332216 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.332269 (+    53us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.332312 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.498974 (+1166662us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.499014 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.499128 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2338310400}
I20240426 13:19:37.647107 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:37.562491 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88394.8us
I20240426 13:19:37.477191 27116 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.478646 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.539319 27061 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.604668 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.601365 26964 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.607715 25099 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.608701 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.609150 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.656003 24954 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:37.609977 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.593132 24851 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.540436 26935 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.612253 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.616578 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:37.659665 24950 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.660143 24955 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:37.619184 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.620679 27013 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.621101 26952 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.621615 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.622196 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.669405 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.623718 27082 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.624485 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.673899 25624 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.624644 24952 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:37.601948 27035 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.625736 26839 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.678937 26839 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.680745 26839 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.587672 26978 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.539113 26656 consensus_queue.cc:1159] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:37.683715 26993 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.903s	user 0.009s	sys 0.002s
I20240426 13:19:37.684305 26993 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.686756 26993 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.645305 27115 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.601073 27096 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.549870 27111 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.632899 24953 rpcz_store.cc:276] 0426 13:19:35.324354 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.306830 (+982476us) service_pool.cc:225] Handling call
0426 13:19:36.327385 (+ 20555us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.665664 (+338279us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.665709 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.665761 (+    52us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.665798 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.541134 (+875336us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.541195 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.541337 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777471360}
W20240426 13:19:37.268433 26799 ts_tablet_manager.cc:1432] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Trace:
0426 13:19:26.765217 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.765367 (+   150us) service_pool.cc:225] Handling call
0426 13:19:27.070272 (+304905us) ts_tablet_manager.cc:903] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
0426 13:19:29.296472 (+2226200us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:19:30.014282 (+717810us) tablet_metadata.cc:641] Metadata flushed
0426 13:19:30.014676 (+   394us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:19:30.181366 (+166690us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:19:30.181821 (+   455us) tablet_replica.cc:205] Creating consensus instance
0426 13:19:30.182267 (+   446us) ts_tablet_manager.cc:958] Replica registered.
0426 13:19:30.182364 (+    97us) inbound_call.cc:173] Queueing success response
0426 13:19:32.446086 (+2263722us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0426 13:19:35.057352 (+2611266us) ts_tablet_manager.cc:1399] Starting tablet replica
0426 13:19:35.104464 (+ 47112us) tablet_replica.cc:259] Starting instrumentation
0426 13:19:35.105013 (+   549us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":25453,"mutex_wait_us":126335,"raft.queue_time_us":149965,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":7022,"tablet-copy.queue_time_us":173608,"tablet-download-pool-.queue_time_us":1911,"tablet-download-pool-.run_cpu_time_us":1411,"tablet-download-pool-.run_wall_time_us":1148283,"thread_start_us":1426950,"threads_started":6}
I20240426 13:19:37.632494 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.555270 24853 rpcz_store.cc:276] 0426 13:19:30.837365 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.837634 (+   269us) service_pool.cc:225] Handling call
0426 13:19:37.489903 (+6652269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:32.050702 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:32.220420 (+169718us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:32.220479 (+    59us) write_op.cc:427] Acquired schema lock
0426 13:19:32.220497 (+    18us) tablet.cc:660] Decoding operations
0426 13:19:32.221589 (+  1092us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:32.221722 (+   133us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:32.221757 (+    35us) tablet.cc:683] Acquiring locks for 9 operations
0426 13:19:32.222642 (+   885us) tablet.cc:699] Row locks acquired
0426 13:19:32.222677 (+    35us) write_op.cc:260] PREPARE: finished
0426 13:19:32.222828 (+   151us) write_op.cc:270] Start()
0426 13:19:32.222967 (+   139us) write_op.cc:275] Timestamp: P: 1714137572222770 usec, L: 0
0426 13:19:32.222997 (+    30us) op_driver.cc:331] REPLICATION: starting
0426 13:19:32.223643 (+   646us) log.cc:842] Serialized 3547 byte log entry
0426 13:19:32.910145 (+686502us) op_driver.cc:444] REPLICATION: finished
0426 13:19:33.306451 (+396306us) write_op.cc:300] APPLY: starting
0426 13:19:33.341234 (+ 34783us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:33.353816 (+ 12582us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:33.359507 (+  5691us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:33.417538 (+ 58031us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:33.417826 (+   288us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0426 13:19:33.417875 (+    49us) write_op.cc:311] APPLY: finished
0426 13:19:33.418217 (+   342us) log.cc:842] Serialized 71 byte log entry
0426 13:19:33.418403 (+   186us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:33.418733 (+   330us) write_op.cc:446] Released schema lock
0426 13:19:33.418807 (+    74us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:33.419068 (+   261us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:33.419095 (+    27us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":788,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":285566,"apply.run_cpu_time_us":4635,"apply.run_wall_time_us":139635,"num_ops":9,"prepare.queue_time_us":446026,"prepare.run_cpu_time_us":17637,"prepare.run_wall_time_us":442319,"raft.queue_time_us":130834,"raft.run_cpu_time_us":24100,"raft.run_wall_time_us":291145,"replication_time_us":612415,"thread_start_us":727136,"threads_started":4,"wal-append.queue_time_us":342193,"wal-append.run_cpu_time_us":10356,"wal-append.run_wall_time_us":1955326}]]}
I20240426 13:19:37.639158 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.639775 25028 rpcz_store.cc:276] 0426 13:19:35.323903 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.331210 (+  7307us) service_pool.cc:225] Handling call
0426 13:19:35.485659 (+154449us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.418467 (+932808us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.418495 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.418531 (+    36us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.418557 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.537396 (+1118839us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.537439 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.537553 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2144953088}
I20240426 13:19:37.642099 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.639111 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.707651 24950 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.601512 26757 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.651325 25097 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:37.712198 25097 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.470999 26859 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.651434 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 172872us
W20240426 13:19:37.628612 27143 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:19:37.668593 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.652283 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.631322 25620 rpcz_store.cc:276] 0426 13:19:35.274076 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.275770 (+  1694us) service_pool.cc:225] Handling call
0426 13:19:35.290593 (+ 14823us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:35.291023 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.291052 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:35.291593 (+   541us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.070786 (+779193us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.680649 (+609863us) log.cc:842] Serialized 14 byte log entry
0426 13:19:37.539685 (+859036us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b540053e420. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902ab143 00007fda9031809f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c
0426 13:19:37.539747 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.539834 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.539876 (+    42us) raft_consensus.cc:1671] finished
0426 13:19:37.539950 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.540077 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":729277,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":32383,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"spinlock_wait_cycles":48510848,"thread_start_us":1388189,"threads_started":2,"wal-append.queue_time_us":866746}
W20240426 13:19:37.653620 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 217752us
I20240426 13:19:37.561703 27084 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.608085 27113 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.657701 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.657738 24954 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:37.662257 24851 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:19:37.723173 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65549.4us
I20240426 13:19:37.724261 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:37.678112 27035 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.702903 27115 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.560057 26945 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.705400 27066 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.661715 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 334) took 3087 ms (client timeout 1581 ms). Trace:
W20240426 13:19:37.707878 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75471.2us
I20240426 13:19:37.738313 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.483629 27136 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.708879 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66864.1us
I20240426 13:19:37.676406 26952 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.744202 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.714998 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.745151 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.745555 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:37.745744 25619 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.746698 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.639174 27141 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.747846 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.719691 24951 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.724107 24851 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.749770 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.750396 24851 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:19:37.751564 26861 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:37.660161 27112 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.750393 26413 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 13:19:37.756940 26413 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:37.757279 26413 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.757711 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:37.758126 26413 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.759479 26413 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.665720 26935 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.731408 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.953s (SENT)
I20240426 13:19:37.711679 26993 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.717213 26839 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.728345 27029 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.761536 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.744578 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.730674 26757 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:37.768352 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:37.752421 27141 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.788448 26757 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:37.788926 26757 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.672737 27013 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.753331 27066 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.753734 27111 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.713222 25097 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:37.729969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127242us
I20240426 13:19:37.757293 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.736904 26945 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:37.736752 25621 rpcz_store.cc:269] 0426 13:19:34.564480 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.564682 (+   202us) service_pool.cc:225] Handling call
0426 13:19:37.636074 (+3071392us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.651458 (+ 15384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3902220160}
I20240426 13:19:37.746961 27085 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.728582 27084 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.761914 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.769727 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.752936 27035 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:37.791857 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62122.2us
I20240426 13:19:37.802246 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.756134 24851 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.793957 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.850466 25030 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.850687 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.851253 25026 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.852084 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.852944 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.762625 24950 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.754277 27113 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.801923 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.856211 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.856832 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:37.842486 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80649us
I20240426 13:19:37.811223 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.860297 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.835016 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.812389 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:37.794658 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.787117 24952 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:37.794658 25031 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:37.866830 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.867550 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:37.796876 25029 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.800349 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.795047 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.795743 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.869661 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.846065 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.795751 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.846724 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.846288 25099 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.871055 25101 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.850128 24851 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.854887 26965 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:37.872402 26965 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.872602 26861 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.872783 26965 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:37.873034 26861 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:37.873917 26861 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.875066 26861 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.876096 27114 consensus_queue.cc:1159] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
W20240426 13:19:37.876103 26996 consensus_queue.cc:1159] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 4 status { last_received { term: 4 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 4 index: 6 } }
I20240426 13:19:37.878139 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.786578 26969 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:37.772881 26857 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.576s	user 0.004s	sys 0.004s
I20240426 13:19:37.800791 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.856707 25022 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.885174 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.882930 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.885193 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.888341 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.898254 25103 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:37.889307 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.780954 27136 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.898236 25102 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:37.901372 24948 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:37.901994 25097 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.793994 27066 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.761823 26894 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.769560 27039 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.895282 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.794370 27102 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.813267 27112 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:37.925473 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:37.904597 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 427844us
I20240426 13:19:37.783622 27096 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.913688 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.928936 25100 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:37.915629 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65078.3us
I20240426 13:19:37.783986 27029 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.778425 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:37.917624 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59212.6us
I20240426 13:19:37.952302 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.885546 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.957083 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.959750 25069 leader_election.cc:398] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
I20240426 13:19:37.923347 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.924108 25022 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.761766 27036 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.152s
I20240426 13:19:37.907347 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.896317 27035 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.771819 26859 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.896324 27111 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.904953 26656 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:37.926108 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:37.939574 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.915642 26962 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.913602 26839 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.879874 26982 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.919687 27088 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20240426 13:19:37.904348 27115 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.961632 27147 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:37.923197 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.913801 27107 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.954779 24851 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:38.036445 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.037781 25101 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.985550 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.998481 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.972214 26413 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20240426 13:19:38.039840 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.987000 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:38.032878 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109652us
I20240426 13:19:37.973958 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.041492 25023 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.041657 25031 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:38.039369 25097 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:38.033006 25069 leader_election.cc:401] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240426 13:19:37.922204 27066 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.045531 25096 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.921590 27039 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:38.057093 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:38.057878 25103 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.058310 25103 raft_consensus.cc:603] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:38.040472 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:38.060827 25027 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.056155 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.062434 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.062815 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.063920 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.064945 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.065104 25025 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.948403 27029 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.065876 25102 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:37.988014 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:38.069017 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:38.069716 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 110207us
I20240426 13:19:38.070010 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:37.962567 27013 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.964519 27112 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.075467 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.044507 26656 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.985709 26935 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.968081 26945 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.968022 26721 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.074867 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.001917 27084 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.033340 26839 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:38.077684 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 311370us
I20240426 13:19:38.061655 27035 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.068652 27096 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.100142 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.100842 24952 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.100858 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.103404 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.036131 27106 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.986550 26952 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.046581 26894 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.046330 27107 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.084830 26413 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:38.079705 27020 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.122195 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.046221 27111 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.087312 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.097908 26721 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.002542 26757 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.979940 26965 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.981194 26747 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.115999 26413 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.947903 26824 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.131s
I20240426 13:19:38.075840 27150 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
W20240426 13:19:38.117275 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 304999us
I20240426 13:19:38.036201 26962 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.082676 26656 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.085026 27036 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:38.068801 25618 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.075764 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309 on tablet 11fad756437243d2959f5998d6bf8309 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:38.132159 26952 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.125892 25098 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.154853 24952 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.128917 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.133164 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.099184 26839 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.081022 27147 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240426 13:19:38.136929 26413 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.162916 26413 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.118402 27035 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.127497 27020 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.123608 27107 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.115113 26773 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.082607 27068 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 13:19:38.086032 27051 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20240426 13:19:38.133666 27061 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:38.148403 25619 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.125864 26977 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.151423 25026 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.149435 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.184098 27096 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.184708 27096 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.185005 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.186614 27096 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.186750 27113 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.187129 27096 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:38.130292 26721 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.098817 27013 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.156724 27036 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.144184 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:38.157745 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.134168 27088 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.188832 27036 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:38.190112 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.159312 25030 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.193084 27084 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:38.193677 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.194449 27084 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.194849 27084 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.195171 27084 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.197296 27111 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.198580 27088 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.175979 27112 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.201701 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.136379 26894 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.202859 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.203871 26757 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.204391 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:38.205546 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 477) took 4654 ms (client timeout 1645 ms). Trace:
I20240426 13:19:38.205499 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.206430 25617 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.159744 25096 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:38.120046 27106 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.191831 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:38.160825 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 304078us
I20240426 13:19:38.209251 27106 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.152796 27114 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.209815 27114 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.210606 27114 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.211557 27114 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.211432 27085 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:38.203805 27051 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.135681 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.215837 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.212215 27111 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.217062 27111 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.217448 27111 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.202883 26894 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.217746 26747 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.212105 26952 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.212433 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.226867 25619 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:38.212798 26912 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:38.185088 26973 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.221336 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:38.206041 25615 rpcz_store.cc:269] 0426 13:19:33.551265 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.658303 (+107038us) service_pool.cc:225] Handling call
0426 13:19:38.205382 (+4547079us) spinlock_profiling.cc:243] Waited 4.33 s on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:38.205464 (+    82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9954013568}
I20240426 13:19:38.203946 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.223012 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.242242 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.241055 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.242725 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.243690 24948 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.248309 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.153832 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.247126 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:38.256413 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 144556us
I20240426 13:19:38.177098 27020 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.166071 27016 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:38.262082 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.262970 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.265271 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.266769 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.187976 27096 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:38.181417 26773 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.272161 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.201735 26962 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.260497 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:38.273722 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.279011 26961 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:38.279165 27152 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20240426 13:19:38.279510 26961 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.272383 27096 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.279968 26961 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.281006 27096 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.282390 27096 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:38.283217 26978 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.602s	user 0.006s	sys 0.002s
I20240426 13:19:38.284317 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:38.284951 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146730us
I20240426 13:19:38.287243 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.245944 26973 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.275182 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:38.257625 27088 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.264593 27113 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.297514 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.257369 27061 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.288151 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.300969 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.272943 27013 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.267915 27068 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.292791 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:38.305881 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.273309 26773 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.310333 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.313536 25096 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:38.289227 27016 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.288722 27115 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.322994 26413 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.323448 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.291168 27038 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:38.320772 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.296368 26973 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.328359 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:38.307281 27085 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:38.297255 27106 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.332978 26413 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.308190 26969 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.283615 26978 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.293306 27112 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.320363 27113 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.361479 27113 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.362323 27113 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.328063 26713 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.365382 26952 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.366586 27061 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.369848 27039 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.371206 27039 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.320484 27088 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.315690 26982 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.353832 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:38.388633 27035 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.323911 27016 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.389338 27016 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.356055 26969 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.390170 26969 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.327936 26985 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:38.392244 26985 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.392632 26985 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.394919 25023 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
W20240426 13:19:38.388525 27159 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20240426 13:19:38.311828 27102 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240426 13:19:38.335219 27038 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.430531 26912 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.430697 27038 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:38.430931 26912 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.431743 26912 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.433112 26912 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.348958 27115 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:38.427651 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73950.8us
I20240426 13:19:38.429056 27085 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.439963 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.440191 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.441556 25096 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.354418 26965 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.460405 26969 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.461428 26945 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.461918 26945 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.462631 26978 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.463043 26945 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.443817 27168 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20240426 13:19:38.439435 26747 consensus_queue.cc:1159] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20240426 13:19:38.466022 26977 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.446247 26985 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.448920 27029 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.466434 26977 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.467319 26977 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.467885 27029 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.454778 26656 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.471792 26656 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.472182 27085 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.472334 26965 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.472896 26965 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.473315 27085 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.460405 26773 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.427945 27160 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:19:38.468240 25620 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.474886 27107 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.468808 27088 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.481081 25031 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.481125 27088 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.483038 27088 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.460137 27020 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.485782 27016 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:38.458295 27136 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s	user 0.008s	sys 0.000s
I20240426 13:19:38.473325 26965 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.515982 27016 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.447533 26973 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.518530 25621 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.457613 26721 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.520805 25104 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.520954 27084 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.352360 26757 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20240426 13:19:38.469736 26985 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.489015 26961 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.443152 27111 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.538372 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.538990 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.539346 27111 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.499131 27066 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.513759 27038 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.536198 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.537330 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:38.550916 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 284229us
I20240426 13:19:38.551648 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.552358 27111 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.552695 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.543185 25096 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:38.553409 27111 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:38.541913 26977 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.553942 27111 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.554977 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:38.560751 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 236390us
I20240426 13:19:38.561141 25624 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.554461 27111 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.562970 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.562918 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:38.565500 25031 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:38.541815 26945 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.558521 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.533684 26969 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.571615 24947 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.541497 26973 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.572840 25023 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.574997 26413 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.602862 26413 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.575496 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:38.539350 26961 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.572345 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:38.604751 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.605506 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.606112 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:38.593444 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 270060us
I20240426 13:19:38.595454 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.598341 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.554847 26978 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20240426 13:19:38.561758 27096 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.603410 26413 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.541509 26773 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:38.599699 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.617923 25104 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.530068 26965 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.568886 27020 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:38.628924 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56661.9us
I20240426 13:19:38.568565 26987 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:38.601249 27016 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.607048 27084 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.632961 26993 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.601732 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.569553 27162 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20240426 13:19:38.645427 27108 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.654476 26413 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.657372 24955 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.641157 26993 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.653591 27108 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.658766 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.683845 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.649967 27036 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:38.628667 27096 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.661446 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 374261us
I20240426 13:19:38.669157 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.670569 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.691900 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.693092 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.694864 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.668982 27112 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.654171 26773 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.674713 25104 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.641515 27016 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.683579 26993 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.686177 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.697959 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.700711 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.679965 26413 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.716192 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.716341 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.717444 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.706897 24950 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:38.707861 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:38.687986 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:38.722047 27170 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20240426 13:19:38.724431 26413 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:38.725554 26413 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.725893 26413 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.726727 26413 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.727753 26413 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.728562 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.729496 24952 raft_consensus.cc:2354] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:38.730193 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.730870 24952 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.731528 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.732802 24952 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.732901 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.734714 25624 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:38.735850 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.736299 26912 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.736703 24950 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.732447 27091 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.738443 27091 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:38.738997 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.739657 24948 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.739240 27091 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.740361 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.701591 27162 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.741142 24951 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
W20240426 13:19:38.740805 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52937.6us
I20240426 13:19:38.750420 25101 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.705046 26773 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.743870 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.706310 27051 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.700647 27150 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.701653 27066 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.701400 27036 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.754168 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.705559 27061 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.757967 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:38.741724 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 453668us
I20240426 13:19:38.745723 26721 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:38.760784 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:38.726902 27141 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.762091 25026 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:38.724833 27146 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.722527 27163 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.781352 24952 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:38.805368 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.782299 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.806360 25624 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.795661 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.808444 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.809280 25105 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:38.810366 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.793274 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.794834 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.794912 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.811419 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.797472 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.812211 25097 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.812211 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.798854 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.798346 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.797168 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.812973 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.813552 25097 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.814076 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.797756 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.797809 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.799046 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.801743 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.826258 25100 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.812542 25031 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.796850 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.820495 27171 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.781414 26965 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.813266 25099 raft_consensus.cc:2362] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6. Re-sending same reply.
W20240426 13:19:38.823781 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63110.8us
I20240426 13:19:38.807898 27036 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.819224 24952 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.820331 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.824316 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.828390 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.828516 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.840283 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.830528 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.840926 25022 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.827653 25620 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.832518 25024 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:38.832442 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.833086 25029 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:38.840516 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.842701 27084 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.844020 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.846827 25097 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:38.846944 27066 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.847468 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.722262 27108 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.856123 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.857507 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.840435 25026 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.860242 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.860952 25029 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.846071 24949 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.862776 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.846120 24952 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.849478 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.866508 24921 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.866798 24921 leader_election.cc:401] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:38.876646 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.877672 24948 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:38.878933 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 78077.3us
I20240426 13:19:38.834589 26985 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.891647 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.895233 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.896633 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.897799 25029 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:38.898691 25027 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.878449 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.866986 27179 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.905004 24948 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:38.906291 24955 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.907377 24949 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.884477 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.888650 27107 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.849164 26773 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.909271 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:38.911777 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63054us
I20240426 13:19:38.912406 24952 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:19:38.914028 24951 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.915391 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.918399 25099 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:38.897061 27108 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.921542 25031 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.888669 27096 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.898823 27066 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.923110 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.923938 24948 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:38.924636 25622 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.923609 25069 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:38.929317 25069 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:38.910532 26656 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.909880 27084 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.916427 27088 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.910233 26985 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:38.967710 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52249.5us
I20240426 13:19:38.968731 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:38.917480 27157 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:38.969141 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45661.6us
I20240426 13:19:38.976131 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.978518 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.904039 27136 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20240426 13:19:38.924976 27112 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.910828 27035 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240426 13:19:38.990202 25102 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:38.848690 27171 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.993213 27106 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:19:38.995620 27175 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20240426 13:19:38.996702 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.978055 26656 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:39.001025 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.919407 27162 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 13:19:39.004670 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:38.993352 27114 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.781s	user 0.014s	sys 0.000s
I20240426 13:19:39.016706 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.033514 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.035410 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.035656 25030 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.928107 27036 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.977658 26978 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.996484 27150 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.930164 27179 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.085s
I20240426 13:19:39.041507 25027 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:39.041934 25031 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 13:19:39.050055 25031 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 1
I20240426 13:19:39.050287 25031 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 13:19:39.050549 25031 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:39.013969 26679 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.006269 27016 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.059409 25030 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:39.013860 27106 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.997684 26757 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20240426 13:19:38.988782 27157 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.033797 27147 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.024627 26983 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:39.034343 27093 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:39.040391 25621 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:39.093371 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 117373us
I20240426 13:19:39.105574 25102 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
W20240426 13:19:39.144591 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 287) took 1380 ms (client timeout 1500 ms). Trace:
W20240426 13:19:39.166352 24947 rpcz_store.cc:269] 0426 13:19:37.764014 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:37.768141 (+  4127us) service_pool.cc:225] Handling call
0426 13:19:39.144340 (+1376199us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54000bf754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:39.144504 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800937728}
I20240426 13:19:39.043851 27187 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.015820 27085 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240426 13:19:39.045439 26996 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.155045 25105 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:39.115233 27192 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
I20240426 13:19:39.052313 27029 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20240426 13:19:39.168881 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:39.121034 27191 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20240426 13:19:39.175455 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.188406 25622 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:39.188992 27136 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:39.158926 27196 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20240426 13:19:39.121349 26679 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.190409 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.045480 27036 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.051862 26978 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.195161 25620 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20240426 13:19:39.197517 27195 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
W20240426 13:19:39.214879 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 124083us
W20240426 13:19:39.198297 27194 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.424s	user 0.005s	sys 0.000s
I20240426 13:19:39.202716 25069 leader_election.cc:398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:39.215780 25069 leader_election.cc:401] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:39.204838 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48646us
I20240426 13:19:39.186501 27107 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:39.213953 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 291104us
I20240426 13:19:39.212572 24920 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
W20240426 13:19:39.282621 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 79293us
I20240426 13:19:39.283200 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.286429 24949 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:39.287304 24951 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.287408 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:39.260372 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57817.8us
I20240426 13:19:39.289319 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.289350 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.291541 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.293318 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.294566 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:39.260900 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43981.1us
I20240426 13:19:39.295871 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.298317 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.299034 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.300626 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.060309 26985 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:39.211855 26839 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.029s	user 0.012s	sys 0.005s
I20240426 13:19:39.192880 26861 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.208896 26996 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.262658 27162 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.209707 27136 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240426 13:19:39.266319 27111 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.692s	user 0.008s	sys 0.000s
I20240426 13:19:39.319243 27111 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.320415 24954 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:39.320824 26945 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.728s	user 0.004s	sys 0.004s
I20240426 13:19:39.186998 27106 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.321427 25104 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:39.158031 26656 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.322503 26965 consensus_queue.cc:1159] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:39.324904 27111 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.260601 27146 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:39.273041 27190 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20240426 13:19:39.325479 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:39.208626 27147 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.327093 27147 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.328969 25103 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:39.273725 26978 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.275571 27171 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.276104 24948 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 13:19:39.277062 27157 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.277400 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.277447 27203 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240426 13:19:39.278621 26679 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.199860 27187 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20240426 13:19:39.338073 27187 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:39.338411 27187 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.339148 27187 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.191679 27093 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:39.263860 27200 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:19:39.314481 27173 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:39.183596 26983 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.262948 26977 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.709s	user 0.010s	sys 0.000s
I20240426 13:19:39.309840 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.219041 27038 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.305078 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.260462 26721 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20240426 13:19:39.303963 27197 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:39.318769 26993 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.585s	user 0.008s	sys 0.000s
I20240426 13:19:39.309057 26839 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.324450 26996 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:39.275184 27193 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.500s	user 0.002s	sys 0.009s
I20240426 13:19:39.303752 27176 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.304173 27141 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.390918 24953 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:39.337317 27157 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.340646 27093 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.371774 27187 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:39.365991 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56259.2us
I20240426 13:19:39.307265 27035 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20240426 13:19:39.303248 27107 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.398258 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:39.388556 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82624.6us
W20240426 13:19:39.301465 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 131662us
W20240426 13:19:39.392249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48057.4us
I20240426 13:19:39.368705 27084 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.372200 27173 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.395864 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:39.390007 27165 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.401099 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.410086 24952 raft_consensus.cc:2362] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5. Re-sending same reply.
I20240426 13:19:39.406960 25104 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.372555 26964 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240426 13:19:39.416043 27165 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:39.416401 27165 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.417076 27165 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.417992 26945 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:39.418247 27165 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:39.418977 26945 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:39.420222 26945 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:39.420861 26945 consensus_queue.cc:1225] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:39.421047 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.421320 26945 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:39.421831 26945 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.393518 27114 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:39.422328 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:39.422577 26945 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.423434 27209 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.424329 26857 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.424819 26857 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.425599 26857 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.387667 26993 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.428314 26993 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.431730 25028 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:39.433497 26824 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:39.434644 26824 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:39.435001 26824 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.436121 26824 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.449800 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.437211 26824 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:39.457422 24947 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:39.394950 27157 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.422601 27114 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.451666 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.460178 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.461195 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.462543 25100 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.449380 26839 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.464499 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:39.465266 26413 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:39.465623 26413 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:39.452340 27036 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.465914 26413 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.455549 27084 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:39.467216 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.468066 27016 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:39.453290 27035 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20240426 13:19:39.459558 26945 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:39.472380 27205 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:19:39.474711 26773 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:39.480257 26713 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:39.481211 24950 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:39.488952 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.492254 26413 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.461072 26983 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:39.498535 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.521569 25031 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:39.506459 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:39.522114 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.522621 26413 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.523576 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.524189 25104 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.530671 25028 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:39.536942 25104 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.538152 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:39.541132 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:39.542076 25620 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:39.542918 27061 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.464534 27146 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20240426 13:19:39.545341 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.546705 27016 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:39.547101 27016 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.547557 27016 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.548884 26861 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:39.497830 26839 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 13:19:39.512230 27209 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240426 13:19:39.462564 27040 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240426 13:19:39.561439 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:39.564726 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.565672 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.520522 26713 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.554723 26993 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:39.592268 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.561288 27150 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:39.550662 27084 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240426 13:19:39.619290 27107 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.619410 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:39.512909 27173 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.620316 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.621941 26965 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.622556 26965 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.585680 26983 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.624240 27184 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.565948 26964 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20240426 13:19:39.622556 27186 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:39.625231 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.625506 27186 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.625648 26993 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.626006 27186 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.626533 26993 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.623736 26965 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.624264 25621 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Advancing to term 2
I20240426 13:19:39.597493 27036 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.600275 27157 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.601509 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.632714 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.588526 27115 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240426 13:19:39.551585 27114 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.607944 26413 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.574468 27038 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.586845 27035 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:39.634632 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:39.637620 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.678210 26413 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:39.681263 26413 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:39.683173 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.695231 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.695856 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.696626 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.697176 24952 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.700644 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.701243 24948 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.703661 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.704396 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:39.704710 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.706105 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.639797 26987 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20240426 13:19:39.631563 27162 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 13:19:39.686251 27035 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.712294 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.712426 27035 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.714680 27035 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.700287 27102 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.720180 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.721097 25101 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:39.722806 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.652199 27187 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:39.723778 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:39.649276 27038 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.727512 25624 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:39.725010 24955 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.733011 24952 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.687326 27107 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:39.761168 25101 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:39.758155 24948 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.764961 26413 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:39.757582 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:39.767081 25624 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.687623 26839 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.645278 27157 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20240426 13:19:39.726512 27187 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:39.832158 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.758646 26969 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.757915 26757 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.760123 27107 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.801762 27187 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:39.839730 26679 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:39.814800 26839 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:39.913674 25624 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.890244 26977 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.892580 26757 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.872093 26969 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.841393 27186 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.940659 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 177882us
W20240426 13:19:39.948661 27223 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
W20240426 13:19:39.948763 27226 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20240426 13:19:39.950556 27221 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:19:39.950932 27102 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:39.952629 26969 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.954350 25029 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:39.955539 25030 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:39.958725 25029 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.961596 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.910825 26679 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.926323 26824 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20240426 13:19:39.967330 26839 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:39.897418 27038 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.987191 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.929298 26977 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.993804 25029 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:40.001050 24956 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:40.001431 24956 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 13:19:40.011101 24956 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:39.926028 27107 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.030079 24948 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20240426 13:19:40.025457 27236 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20240426 13:19:40.000077 27225 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.105s	user 0.001s	sys 0.001s
W20240426 13:19:40.009028 27224 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
W20240426 13:19:40.036829 26945 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.516s	user 0.002s	sys 0.006s
W20240426 13:19:40.037693 27176 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.000s
I20240426 13:19:40.037998 27186 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.038101 27176 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:40.039598 25102 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
W20240426 13:19:40.024819 27241 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20240426 13:19:39.977627 27163 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240426 13:19:40.018312 27244 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:40.020519 27228 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20240426 13:19:39.988530 27035 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240426 13:19:40.052822 25103 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:40.055354 25100 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:40.024602 27016 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.060220 27016 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.029547 26773 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 13:19:40.066218 25024 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:40.066810 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:40.068398 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.070688 25028 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:40.071836 25022 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.073042 24948 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:40.071328 25023 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.076783 24948 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:40.077270 27147 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:40.082778 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:40.080777 27147 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:40.086572 26977 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.057488 26757 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.038723 27246 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240426 13:19:40.099450 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:40.100983 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:40.108777 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.109426 25624 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:40.110419 25103 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:40.110755 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.113622 26993 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:40.113754 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:40.114208 26993 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:40.115048 27242 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:19:40.115226 26993 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.051291 27244 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.112085 27186 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:40.106194 27147 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.314445 27190 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20240426 13:19:40.332366 27244 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:40.300244 27234 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20240426 13:19:40.109087 27038 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.116888 26679 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:40.312952 27175 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
W20240426 13:19:40.302242 27257 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20240426 13:19:40.314697 27187 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.362376 27038 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:40.368424 27061 consensus_queue.cc:1159] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 4 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:40.368261 27147 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.383390 27051 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:40.387676 27051 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:40.388594 27051 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.389791 27038 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.385749 27187 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.469231 27268 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.492291 27268 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.472649 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.474835 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.494279 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:40.494488 24950 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:40.474853 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:40.487587 27271 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.471102 27269 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.495887 27268 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.502979 25027 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.503180 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.518816 25024 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:40.504619 25026 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:40.500260 27269 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.499876 27271 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.515730 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:40.545704 26679 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:40.566243 27035 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.567597 27271 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.567379 27035 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.569758 27268 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.570447 27106 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.570519 27035 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.574273 27171 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.575059 27016 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.546852 26679 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.570776 27268 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.146124 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.623958 27268 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.625458 26964 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.626592 26964 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.628232 26839 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.629298 26839 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.637156 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.638338 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:40.638903 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.639101 27102 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.644774 25104 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:40.646437 27238 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.728s	user 0.012s	sys 0.006s
I20240426 13:19:40.648520 27269 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.652590 27248 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:40.653108 27248 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.653553 27248 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.654847 27102 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.660221 26713 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.662062 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 146795us
I20240426 13:19:40.662438 27253 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:40.663002 27253 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.661502 27102 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.665097 27253 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:40.719841 27260 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.720791 27260 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:40.723925 27232 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20240426 13:19:40.735841 27016 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:40.723961 27260 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.744302 25022 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:40.748090 26824 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.748800 26824 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.749424 26679 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.749907 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:40.750901 26824 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.716770 26839 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.711954 27171 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.755070 24951 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:40.738852 27016 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.758872 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:40.715386 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 197504us
I20240426 13:19:40.768436 25100 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:40.768733 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.769675 25022 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:40.780623 25105 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.666s	user 0.000s	sys 0.013s
I20240426 13:19:40.781258 25105 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:19:40.793641 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.005s	sys 0.029s
W20240426 13:19:40.789479 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.019s	sys 0.017s
W20240426 13:19:40.797886 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.006s	sys 0.029s
I20240426 13:19:40.798151 25031 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
W20240426 13:19:40.801822 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.019s	sys 0.018s
W20240426 13:19:40.800347 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.004s	sys 0.048s
W20240426 13:19:40.804565 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.005s	sys 0.048s
W20240426 13:19:40.806461 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 705533us
W20240426 13:19:40.806614 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50771.6us
I20240426 13:19:40.809211 27282 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.809929 27282 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.811556 24950 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:40.814011 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:40.818928 27038 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.821228 27085 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:40.821331 26679 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:40.825524 27020 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:40.868362 25104 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:40.826270 27093 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:40.830178 27171 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.871078 26824 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.871527 26824 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.871601 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:40.872962 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58334.3us
I20240426 13:19:40.874419 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:40.874568 27282 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.839150 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.876698 25624 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:40.866431 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52498.8us
I20240426 13:19:40.877362 27038 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.877977 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:40.878273 25100 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:19:40.840605 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.879153 27284 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:40.879796 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.879987 27284 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:40.881422 27016 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:40.871897 26824 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.881953 27016 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:40.883109 27016 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.883337 27248 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.884186 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80390.8us
I20240426 13:19:40.884521 27165 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.885200 27165 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:40.885677 27248 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.886524 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:40.887202 27165 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.887975 25103 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:40.889230 25024 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:40.889461 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.890018 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.890123 24947 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:40.869315 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 752518us
I20240426 13:19:40.846232 27085 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:40.897677 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.900600 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.894246 27248 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.902001 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:40.896735 26679 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:40.903082 26679 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:40.903493 26679 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:40.905884 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.897838 27253 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.912329 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.913215 25022 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.914707 27085 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.915958 27085 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.920563 25100 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:40.920923 26679 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.923362 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:40.923485 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.925388 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.926236 25028 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.926856 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.932590 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.895247 27165 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.926221 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.936079 24952 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:40.925711 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:40.942070 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 134197us
I20240426 13:19:40.944478 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:40.946888 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.949995 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.951216 27150 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:40.951645 27150 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.952047 27150 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:40.948747 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.943672 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:40.957954 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.959215 27115 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.959700 27115 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.960225 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.447824 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:40.960106 27115 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.962162 27165 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.966449 27141 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.967118 27141 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.968207 25031 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.969125 27141 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.971994 27106 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.972223 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1368) took 1252 ms. Trace:
I20240426 13:19:40.972741 25027 rpcz_store.cc:276] 0426 13:19:39.719687 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.722575 (+  2888us) service_pool.cc:225] Handling call
0426 13:19:40.971990 (+1249415us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:40.972136 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1791739136}
I20240426 13:19:40.962510 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.975317 27276 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:40.977037 27146 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.977710 27146 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.980978 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.981334 27163 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.981680 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:40.981951 27163 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.982893 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:40.983060 25100 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:40.984184 27253 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.986845 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.972779 27106 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.930358 27248 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.997704 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:40.997095 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:41.009222 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.009577 27186 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:41.012681 27108 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.012799 27162 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:41.013394 27162 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.014047 27162 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:41.013461 27108 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.016876 25029 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:41.026093 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84308us
I20240426 13:19:40.994782 27146 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.994623 27163 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.971999 27165 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.054366 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.047976 27276 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:41.060134 24954 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:41.062885 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:41.063067 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.059616 27141 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.064118 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.064474 27141 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:41.066251 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1493) took 2095 ms (client timeout 2206 ms). Trace:
I20240426 13:19:41.064888 25023 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:41.067790 27141 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.240706 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.241434 27106 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.078634 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:41.191884 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 133488us
I20240426 13:19:41.228797 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1519) took 1083 ms. Trace:
I20240426 13:19:41.243557 27107 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:41.243700 25103 rpcz_store.cc:276] 0426 13:19:40.145482 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.145806 (+   324us) service_pool.cc:225] Handling call
0426 13:19:41.228534 (+1082728us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400384fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:41.228714 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":511661824}
I20240426 13:19:41.061534 26679 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.245862 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 186121us
I20240426 13:19:41.245666 27150 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:41.235751 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 284) took 1178 ms. Trace:
I20240426 13:19:41.227355 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.269857 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:41.271800 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 202154us
I20240426 13:19:41.074929 27084 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:41.283855 27296 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.187s	user 0.003s	sys 0.000s
I20240426 13:19:41.283689 27084 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.266973 25026 rpcz_store.cc:276] 0426 13:19:40.057397 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.057559 (+   162us) service_pool.cc:225] Handling call
0426 13:19:40.074637 (+ 17078us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:40.520744 (+446107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.235405 (+714661us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.235654 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986973056}
I20240426 13:19:41.093461 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.168720 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:41.307718 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.077598 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:41.166956 27283 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20240426 13:19:41.077595 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.314752 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.232686 27272 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.321306 24947 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:41.321975 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1469) took 1205 ms. Trace:
I20240426 13:19:41.322533 24948 rpcz_store.cc:276] 0426 13:19:40.079998 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.080184 (+   186us) service_pool.cc:225] Handling call
0426 13:19:40.967761 (+887577us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:40.967976 (+   215us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.968322 (+   346us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.968346 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.968787 (+   441us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.011010 (+ 42223us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:41.011050 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.011113 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.282960 (+271847us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:41.285113 (+  2153us) raft_consensus.cc:1671] finished
0426 13:19:41.285138 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.285275 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6558,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":190607488,"thread_start_us":37995,"threads_started":1,"wal-append.queue_time_us":127724}
I20240426 13:19:41.074748 26721 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.324955 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.325567 26721 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.322046 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.328750 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1508) took 2035 ms. Trace:
I20240426 13:19:41.329231 26979 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.329118 25102 rpcz_store.cc:276] 0426 13:19:39.292994 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.299361 (+  6367us) service_pool.cc:225] Handling call
0426 13:19:39.306115 (+  6754us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:40.039962 (+733847us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.313049 (+1273087us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.328579 (+ 15530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1686546304}
I20240426 13:19:41.329818 26979 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.331457 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:41.327942 26721 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.332546 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1736) took 1723 ms. Trace:
I20240426 13:19:41.333117 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1821) took 1426 ms. Trace:
I20240426 13:19:41.332230 27176 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.096372 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:41.333843 27176 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.334511 25615 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.339288 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 497) took 1842 ms. Trace:
I20240426 13:19:41.233207 27248 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.321375 27281 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.230s	user 0.011s	sys 0.050s
I20240426 13:19:41.331748 26979 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.316488 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.332047 25022 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:41.096457 27267 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:41.066687 25099 rpcz_store.cc:269] 0426 13:19:38.970283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:38.970430 (+   147us) service_pool.cc:225] Handling call
0426 13:19:41.065904 (+2095474us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:41.066041 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2235574144}
I20240426 13:19:41.092267 27093 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.181257 26964 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.441031 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:41.442255 26964 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.443259 27084 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.442965 26964 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.445621 27248 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.446170 27248 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.445680 27210 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.448300 26965 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.448360 27210 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.449120 26965 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.450214 26679 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.451172 26965 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.452601 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.453172 27210 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.073036 26945 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.454329 27248 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.454254 26945 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.455873 27291 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:41.455984 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 85641.5us
I20240426 13:19:41.456396 26945 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.456539 27291 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.321867 27272 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.458701 27291 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:41.441097 26824 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.559s	user 0.008s	sys 0.002s
I20240426 13:19:41.459203 27291 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.333559 25030 rpcz_store.cc:276] 0426 13:19:39.906360 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.906498 (+   138us) service_pool.cc:225] Handling call
0426 13:19:39.924880 (+ 18382us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:39.972431 (+ 47551us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.332713 (+1360282us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.332967 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69971200}
I20240426 13:19:41.068090 27061 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.424s
I20240426 13:19:41.450837 26679 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.167688 27271 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.402693 25031 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:41.335937 27176 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:41.465669 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 157028us
I20240426 13:19:41.333254 27288 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.195s
I20240426 13:19:41.459546 26824 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.332921 24953 rpcz_store.cc:276] 0426 13:19:39.562492 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.562662 (+   170us) service_pool.cc:225] Handling call
0426 13:19:39.562982 (+   320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:39.572454 (+  9472us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:40.652952 (+1080498us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b180029bf60. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:19:40.800017 (+147065us) log.cc:842] Serialized 14 byte log entry
0426 13:19:40.961996 (+161979us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:41.007658 (+ 45662us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.007814 (+   156us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:41.008384 (+   570us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.048190 (+ 39806us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.056743 (+  8553us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.273312 (+216569us) raft_consensus.cc:1671] finished
0426 13:19:41.273344 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.285657 (+ 12313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62250,"prepare.queue_time_us":345511,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":65690,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":22737,"spinlock_wait_cycles":5281152,"thread_start_us":200482,"threads_started":2,"wal-append.queue_time_us":262441}
I20240426 13:19:41.490797 25615 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:41.474143 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.307103 27162 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.475545 24956 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:41.340085 27282 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.225s
I20240426 13:19:41.340917 27106 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:41.471529 27298 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.373s	user 0.002s	sys 0.000s
I20240426 13:19:41.544664 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:42.365685 27111 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.938326 27276 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:42.258525 27016 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.566051 27171 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:41.458701 27272 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.772064 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.052s	sys 0.085s
W20240426 13:19:41.470700 27289 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.413s	user 0.002s	sys 0.000s
W20240426 13:19:41.565595 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 404191us
I20240426 13:19:42.371292 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 324) took 1610 ms. Trace:
I20240426 13:19:42.371641 24955 rpcz_store.cc:276] 0426 13:19:40.760669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.760817 (+   148us) service_pool.cc:225] Handling call
0426 13:19:40.761261 (+   444us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.761774 (+   513us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.761808 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.762275 (+   467us) log.cc:842] Serialized 26 byte log entry
0426 13:19:40.791142 (+ 28867us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:40.791192 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:40.791270 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.469991 (+678721us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:42.370920 (+900929us) raft_consensus.cc:1671] finished
0426 13:19:42.370941 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.371133 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1588,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":7772,"spinlock_wait_cycles":4600576,"thread_start_us":28384,"threads_started":1,"wal-append.queue_time_us":170727}
I20240426 13:19:41.564731 26977 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.373473 26977 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.367837 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:42.150417 27280 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.772274 27271 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.374946 26912 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.549755 27114 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:42.377564 27280 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.568362 26965 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.469008 27093 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.568578 27186 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.412506 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.562366 27108 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.550446 27267 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.572803 27051 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.549577 27203 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.567904 27273 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:42.149596 27107 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.893s	user 0.007s	sys 0.004s
I20240426 13:19:41.565090 27284 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:41.999868 27247 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.369027 27016 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.367503 27111 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.459823 27291 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.905151 26979 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.833771 27252 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:42.465373 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 900176us
I20240426 13:19:42.387454 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1748) took 1495 ms. Trace:
I20240426 13:19:41.832824 27136 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.388657 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:41.753063 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.162942 27176 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:42.369918 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.078s	sys 0.114s
I20240426 13:19:42.393291 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:42.368404 27085 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:42.367357 27276 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:41.548501 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:41.777643 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.403708 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1753) took 1318 ms. Trace:
W20240426 13:19:41.735208 27115 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.758s	user 0.001s	sys 0.008s
I20240426 13:19:42.405014 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.473104 27150 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.366221 26713 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:42.411464 25102 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
W20240426 13:19:42.412395 27113 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.069s	user 0.014s	sys 0.014s
I20240426 13:19:42.403168 26679 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.404851 27171 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.420930 24951 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:42.428282 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.369307 27106 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:42.420269 27038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.970s	user 0.041s	sys 0.078s
I20240426 13:19:42.418402 27108 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:42.496239 25022 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.784s	user 0.005s	sys 0.006s
I20240426 13:19:42.523455 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.524015 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1842) took 1203 ms. Trace:
I20240426 13:19:42.524446 25022 rpcz_store.cc:276] 0426 13:19:41.320560 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.320728 (+   168us) service_pool.cc:225] Handling call
0426 13:19:42.523958 (+1203230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:42.523087 27038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s	user 0.042s	sys 0.078s
I20240426 13:19:42.498042 24947 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:42.525187 27108 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.525382 27136 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.526125 26839 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.526702 26839 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.527257 27108 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.527464 27136 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.419649 27272 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.501300 24956 rpcz_store.cc:276] 0426 13:19:41.075690 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.077226 (+  1536us) service_pool.cc:225] Handling call
0426 13:19:42.393973 (+1316747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:42.412951 27186 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:42.507148 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 959108us
W20240426 13:19:42.514879 27113 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.180s	user 0.014s	sys 0.014s
I20240426 13:19:42.509143 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1383) took 1563 ms. Trace:
I20240426 13:19:42.466076 24949 rpcz_store.cc:276] 0426 13:19:40.892010 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.892177 (+   167us) service_pool.cc:225] Handling call
0426 13:19:40.896177 (+  4000us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.931587 (+ 35410us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.931615 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.954549 (+ 22934us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.057285 (+102736us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:41.330360 (+273075us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.470871 (+140511us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:41.564841 (+ 93970us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.564922 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:42.387179 (+822257us) raft_consensus.cc:1671] finished
0426 13:19:42.387279 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.387400 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":452436,"prepare.queue_time_us":273102,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":235489,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":6362,"spinlock_wait_cycles":151641984,"thread_start_us":80591,"threads_started":1,"wal-append.queue_time_us":121204}
I20240426 13:19:42.501332 25099 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:42.518275 25097 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:42.502900 27107 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.527834 27108 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:42.531689 24947 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:42.547432 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 46035.9us
I20240426 13:19:42.461361 27284 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.530366 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.529515 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:42.539995 26839 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.544974 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.549655 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.495260 27247 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.549916 27171 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.434736 26912 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.500921 27273 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.511132 27252 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.514530 26979 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.535501 25028 rpcz_store.cc:276] 0426 13:19:40.945826 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.945928 (+   102us) service_pool.cc:225] Handling call
0426 13:19:41.246708 (+300780us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:41.246951 (+   243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:41.247313 (+   362us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:41.247663 (+   350us) log.cc:842] Serialized 14 byte log entry
0426 13:19:42.508716 (+1261053us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:42.508786 (+    70us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:42.508853 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.508937 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.509079 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1288,"prepare.queue_time_us":750,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"spinlock_wait_cycles":3084544,"thread_start_us":1249864,"threads_started":1}
I20240426 13:19:42.566063 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.528493 27150 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.515565 27115 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.549497 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:42.537304 27259 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.532013 26679 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.579337 27186 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.580216 26964 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.529829 27176 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.583518 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1523) took 1127 ms. Trace:
I20240426 13:19:42.608340 25102 rpcz_store.cc:276] 0426 13:19:41.455869 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.460499 (+  4630us) service_pool.cc:225] Handling call
0426 13:19:41.469322 (+  8823us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.518197 (+1048875us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.583289 (+ 65092us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:42.583464 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2164877696}
I20240426 13:19:42.528038 27136 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.587172 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.587442 27150 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.589154 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.513074 27203 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.549721 26969 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.592752 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.594521 27093 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.594579 27115 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.594892 27174 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:42.595562 25104 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:42.613250 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:42.596422 27291 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.596597 26979 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.596648 27171 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.550071 26656 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.607623 25622 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:42.609853 24952 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.548408 27107 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.589655 27108 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.597170 27259 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.618254 27150 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.617950 27136 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.594662 26713 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.566948 27272 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.615489 26964 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.500155 26965 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.608520 27162 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.638377 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.615865 26945 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.616364 26679 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.643823 26979 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.647825 24949 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:42.659145 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.636076 27186 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.643328 27171 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.639765 27271 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.646920 27174 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:42.681332 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:42.672859 24954 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:42.651170 27197 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.637992 26996 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20240426 13:19:42.615926 26969 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.692852 24954 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:41.339823 25621 rpcz_store.cc:276] 0426 13:19:39.496518 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.501887 (+  5369us) service_pool.cc:225] Handling call
0426 13:19:39.591094 (+ 89207us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.546011 (+954917us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.546079 (+    68us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.546940 (+   861us) log.cc:842] Serialized 26 byte log entry
0426 13:19:40.786955 (+240015us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:40.902593 (+115638us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.020045 (+117452us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.051032 (+ 30987us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.338869 (+287837us) raft_consensus.cc:1671] finished
0426 13:19:41.338904 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.339081 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66431,"prepare.queue_time_us":150233,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":233086,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":253,"spinlock_wait_cycles":9202560,"thread_start_us":666715,"threads_started":3,"wal-append.queue_time_us":389498,"wal-append.run_cpu_time_us":7733,"wal-append.run_wall_time_us":1395081}
I20240426 13:19:42.668982 26839 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.672299 27162 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.686749 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.697295 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:42.690054 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81912us
I20240426 13:19:42.693987 25023 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:42.684295 24947 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:42.697945 25615 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:42.668483 27113 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.580322 27280 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.593401 27247 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.711854 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:42.691217 27174 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.610374 27267 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.670181 26713 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.692215 27016 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.680968 26656 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.720114 27273 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.721159 27136 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.695935 27186 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.716248 27106 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.722615 27176 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.718266 27174 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:42.723378 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.725039 27085 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.726581 25100 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:42.685076 26979 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.728312 25616 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:42.729409 26969 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.732105 27162 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.695537 27271 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.739423 27302 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.733726 27272 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.718626 27113 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.691130 27111 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.673038 27108 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.739449 25105 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:42.682402 26945 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.716352 27280 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.765762 27106 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.852164 27106 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.696475 26964 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.732908 27259 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.763247 26987 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:42.704110 27171 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.820734 25022 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:42.824955 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:42.709578 26839 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.767756 27252 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.762259 26912 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.774960 27203 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.716719 27197 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.937109 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:43.055207 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 118236us
I20240426 13:19:42.987574 26679 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.056104 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.056779 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:42.952564 27107 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:42.992635 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1360) took 2070 ms (client timeout 2129 ms). Trace:
I20240426 13:19:42.781533 27085 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.786365 26996 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.060882 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 360) took 1742 ms. Trace:
I20240426 13:19:43.061568 26839 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.057297 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:43.061803 25624 rpcz_store.cc:276] 0426 13:19:41.318056 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.320839 (+  2783us) service_pool.cc:225] Handling call
0426 13:19:41.415715 (+ 94876us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:42.461141 (+1045426us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.461183 (+    42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:42.520736 (+ 59553us) log.cc:842] Serialized 52 byte log entry
0426 13:19:42.958410 (+437674us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:42.958465 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.958564 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.060653 (+102089us) raft_consensus.cc:1671] finished
0426 13:19:43.060672 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.060819 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":925138,"raft.queue_time_us":998797,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":36130,"thread_start_us":391994,"threads_started":1,"wal-append.queue_time_us":524477}
I20240426 13:19:43.063162 27085 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.058684 27106 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.063235 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.064030 26996 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.064616 25100 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:42.858731 27302 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.066291 26996 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:42.962352 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 316634us
I20240426 13:19:43.066901 27302 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.056861 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:43.056798 26679 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:43.069383 27197 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.069810 26679 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:43.070472 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.047s	sys 0.054s
W20240426 13:19:43.070912 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.047s	sys 0.054s
I20240426 13:19:43.070860 26679 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.068621 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:42.901000 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.000512 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:43.074040 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 173134us
W20240426 13:19:43.074802 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74399.4us
I20240426 13:19:43.074946 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:43.061264 25104 rpcz_store.cc:269] 0426 13:19:40.921837 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.932314 (+ 10477us) service_pool.cc:225] Handling call
0426 13:19:42.992404 (+2060090us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:42.992560 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3810614528}
I20240426 13:19:43.075837 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:42.828593 27253 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.089686 25104 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:43.071759 27106 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.088866 27305 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.058223 26979 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.858469 26987 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:42.951895 27176 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.088913 27162 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.072091 26912 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.120200 27113 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.129215 26912 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.129781 26977 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.089612 27272 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:43.120177 27147 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:43.120910 27114 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:43.143929 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:43.121068 27300 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.424s	user 0.000s	sys 0.000s
I20240426 13:19:43.121181 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.080379 26839 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:43.128764 27162 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:43.073313 27186 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.130378 26656 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.140388 27252 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.141167 27093 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:43.141917 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 73366us
I20240426 13:19:42.937624 27271 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.901597 27280 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.131153 27106 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.242008 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.242846 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:43.243362 27271 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:43.245306 27301 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.550s	user 0.000s	sys 0.001s
I20240426 13:19:43.245589 27280 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.246160 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.185564 27114 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.247819 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1482) took 1884 ms. Trace:
W20240426 13:19:43.225994 25097 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.685s	user 0.005s	sys 0.003s
I20240426 13:19:43.180274 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.250985 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 176147us
I20240426 13:19:43.180429 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.252485 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.248654 25097 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:43.221151 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:43.120935 27061 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:43.187500 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.254734 25097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1525) took 1505 ms (client timeout 1898 ms). Trace:
I20240426 13:19:43.187475 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.256851 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
W20240426 13:19:43.257535 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 113721us
I20240426 13:19:43.254294 27249 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.259370 27252 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.259477 26987 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.260354 27249 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.260844 26987 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.261651 27252 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.262368 27249 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:43.255636 25097 rpcz_store.cc:269] 0426 13:19:41.749367 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.749643 (+   276us) service_pool.cc:225] Handling call
0426 13:19:43.254668 (+1505025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:43.106979 27253 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.255919 25027 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:43.113312 27107 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.263666 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:43.248184 24950 rpcz_store.cc:276] 0426 13:19:41.363364 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.634050 (+1270686us) service_pool.cc:225] Handling call
0426 13:19:42.648585 (+ 14535us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:42.661676 (+ 13091us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.661702 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:42.684885 (+ 23183us) log.cc:842] Serialized 26 byte log entry
0426 13:19:43.015838 (+330953us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:43.015890 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.015969 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.247601 (+231632us) raft_consensus.cc:1671] finished
0426 13:19:43.247634 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.247758 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12731,"raft.queue_time_us":32283,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":11312,"thread_start_us":318653,"threads_started":1,"wal-append.queue_time_us":424642}
I20240426 13:19:43.247211 27162 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.271360 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:43.259317 26964 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.173058 26979 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.272717 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.172878 27038 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:43.224262 27171 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.164111 26969 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.142855 27272 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.187507 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.220052 26713 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.254127 26839 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:43.188937 27303 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.263219 26977 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:43.283931 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.292061 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:43.263830 27106 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.295622 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.293665 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.297379 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.265470 27016 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.299160 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.268201 27108 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.300211 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:43.299870 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 224119us
I20240426 13:19:43.300850 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.269922 27162 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.282521 27271 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:43.304747 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:43.376348 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1760) took 1474 ms. Trace:
I20240426 13:19:43.270323 27249 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
W20240426 13:19:43.377287 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104100us
I20240426 13:19:43.164072 27176 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.164207 27147 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.379400 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.291082 27311 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:43.262869 27093 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.292786 26979 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.301281 27272 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:43.294571 27107 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.379112 26969 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.382311 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.383026 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.384761 24952 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:43.380896 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:43.385434 26977 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:43.385969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94000us
I20240426 13:19:43.386090 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.395457 24949 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:43.417161 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:43.420424 24950 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.420325 24953 rpcz_store.cc:276] 0426 13:19:41.902285 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.908524 (+  6239us) service_pool.cc:225] Handling call
0426 13:19:43.304278 (+1395754us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:43.304454 (+   176us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.304647 (+   193us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.375996 (+ 71349us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.376058 (+    62us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:43.376097 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.376177 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.376290 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18335104}
I20240426 13:19:43.122990 27305 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.375891 26839 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.444839 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.451958 26912 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.473177 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:43.489907 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.536320 25027 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.536513 24947 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:19:43.453132 27276 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.929s	user 0.010s	sys 0.000s
W20240426 13:19:43.546634 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 118141us
I20240426 13:19:43.422837 27271 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:43.550505 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133476us
I20240426 13:19:43.556160 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.572281 27271 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:43.572944 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.573786 25031 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:43.574894 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.577244 27174 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.814s	user 0.005s	sys 0.005s
I20240426 13:19:43.578028 27174 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.532869 26977 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.580789 26977 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.583087 27107 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.504789 27114 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.584853 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.557518 27305 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.558192 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.489825 27311 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.560636 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.565024 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.552258 27162 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.588241 27176 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.506045 27147 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:43.591362 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 327421us
I20240426 13:19:43.595858 24950 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:43.602527 25025 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:43.602535 27303 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.610208 27303 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.611500 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.572121 27203 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:43.610941 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.596877 27197 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.458382 27016 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.611111 27174 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.636968 27176 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240426 13:19:43.641469 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:43.589280 27276 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.647855 27311 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:43.651109 27016 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.656311 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.657294 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.657927 27323 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:43.665287 27284 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.666805 26979 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.668395 27107 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:43.779592 27176 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:43.669209 26969 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.616701 27203 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.780416 27203 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:43.739826 25069 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.808s (SENT) [suppressed 14 similar messages]
W20240426 13:19:43.777554 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 120344us
I20240426 13:19:43.829264 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.833002 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:43.825892 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 548) took 2881 ms (client timeout 3135 ms). Trace:
W20240426 13:19:43.830034 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 229205us
I20240426 13:19:43.837324 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:43.836973 25622 rpcz_store.cc:269] 0426 13:19:40.946922 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.949712 (+  2790us) service_pool.cc:225] Handling call
0426 13:19:43.825680 (+2875968us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:43.825818 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3789052160}
I20240426 13:19:43.828941 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:43.841907 25104 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:43.842213 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:43.842733 25029 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:43.785431 27162 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.640426 27197 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:43.786535 26987 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.001s
I20240426 13:19:43.819097 26977 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.829170 27108 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:43.779646 27107 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.860714 25589 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:43.860821 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.861248 25589 leader_election.cc:401] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:43.834066 27176 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.861557 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:43.847360 26679 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:43.843763 27147 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:43.864478 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1761) took 1937 ms (client timeout 1864 ms). Trace:
W20240426 13:19:43.866854 24955 rpcz_store.cc:269] 0426 13:19:41.926896 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.373271 (+446375us) service_pool.cc:225] Handling call
0426 13:19:43.860672 (+1487401us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:43.864399 (+  3727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13966080}
I20240426 13:19:43.840498 27093 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.867985 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.815289 27272 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.820947 26839 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.877429 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:43.863698 27107 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:43.863394 27108 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.867748 27147 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.884271 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.873346 27093 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.890627 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.899096 26979 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:43.898954 27327 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:43.899662 26979 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:43.900533 26979 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.913385 27272 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.920195 27176 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.922617 27197 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.925787 27271 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.895606 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1370) took 1122 ms. Trace:
W20240426 13:19:43.935616 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 195835us
I20240426 13:19:43.936770 25099 rpcz_store.cc:276] 0426 13:19:42.772832 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.871019 (+1098187us) service_pool.cc:225] Handling call
0426 13:19:43.885220 (+ 14201us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.895163 (+  9943us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.895206 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.895268 (+    62us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.895316 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.895397 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.895538 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:43.935698 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 372) took 1198 ms. Trace:
I20240426 13:19:43.943604 25097 rpcz_store.cc:276] 0426 13:19:42.737777 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.268382 (+530605us) service_pool.cc:225] Handling call
0426 13:19:43.268670 (+   288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.290983 (+ 22313us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.291002 (+    19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:43.380686 (+ 89684us) log.cc:842] Serialized 26 byte log entry
0426 13:19:43.745802 (+365116us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:43.746202 (+   400us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.746424 (+   222us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.935458 (+189034us) raft_consensus.cc:1671] finished
0426 13:19:43.935512 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.935642 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1366,"raft.queue_time_us":3226,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6593,"thread_start_us":291981,"threads_started":1,"wal-append.queue_time_us":416869}
I20240426 13:19:43.889652 27147 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.951517 25103 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:43.891583 27323 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:43.874902 26912 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20240426 13:19:43.957032 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:43.960587 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1527) took 2163 ms. Trace:
I20240426 13:19:43.960966 26977 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.946343 26987 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:43.966265 24996 connection.cc:232] RPC call timeout handler was delayed by -1.2130351069999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
I20240426 13:19:43.966105 25103 rpcz_store.cc:276] 0426 13:19:41.788474 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.809726 (+ 21252us) service_pool.cc:225] Handling call
0426 13:19:43.947023 (+2137297us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:43.947282 (+   259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.951967 (+  4685us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.952221 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16720768}
W20240426 13:19:43.967644 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 137923us
I20240426 13:19:43.967999 25622 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.976663 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.979771 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.889866 27305 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20240426 13:19:43.982122 25096 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:43.967664 27271 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:43.986476 27299 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.289s	user 0.001s	sys 0.008s
I20240426 13:19:43.988313 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.988920 25097 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:43.989694 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:43.990809 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 719501us
I20240426 13:19:43.961560 27108 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.991935 27273 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:43.962240 27323 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:43.998340 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1526) took 2203 ms (client timeout 2585 ms). Trace:
I20240426 13:19:43.990831 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.999994 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.003428 25103 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:44.001892 25096 rpcz_store.cc:269] 0426 13:19:41.780456 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.780582 (+   126us) service_pool.cc:225] Handling call
0426 13:19:43.976491 (+2195909us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:43.983932 (+  7441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45856640}
I20240426 13:19:43.967119 27276 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.003049 25097 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:44.011500 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79029.9us
I20240426 13:19:44.003674 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.019289 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.019956 24950 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:44.020934 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 469220us
I20240426 13:19:44.021056 25029 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:44.021948 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:44.025400 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.025478 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.998034 27273 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:44.026551 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.025765 27310 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.000684 27093 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:44.013440 27323 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:44.025439 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.025640 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.028285 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.026362 24953 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.055855 24950 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.033404 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.035418 27171 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.058192 27171 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:44.058358 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.058640 27171 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:44.036101 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.038058 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.039718 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.040148 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.040745 27305 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.041097 27271 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:44.043431 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.027837 26656 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.044606 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.031635 27273 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:44.045689 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1491) took 1341 ms. Trace:
W20240426 13:19:44.045702 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1751) took 2970 ms (client timeout 1808 ms). Trace:
I20240426 13:19:44.049804 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1369) took 1287 ms. Trace:
I20240426 13:19:44.051276 26824 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.057219 24949 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:44.070878 25027 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.516s	user 0.007s	sys 0.002s
I20240426 13:19:44.220849 25028 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20240426 13:19:44.183204 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146588us
I20240426 13:19:44.227485 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.249737 25031 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:44.222651 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1368) took 1469 ms. Trace:
W20240426 13:19:44.249262 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 209622us
I20240426 13:19:44.155021 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1386) took 1012 ms. Trace:
I20240426 13:19:44.104657 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.262477 25026 rpcz_store.cc:276] 0426 13:19:43.142243 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.164277 (+ 22034us) service_pool.cc:225] Handling call
0426 13:19:43.164653 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.556761 (+392108us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.556791 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.556839 (+    48us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.556871 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.154786 (+597915us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.154835 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.154965 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":901404416}
I20240426 13:19:44.265938 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:44.248425 24951 rpcz_store.cc:269] 0426 13:19:41.075123 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.093151 (+ 18028us) service_pool.cc:225] Handling call
0426 13:19:44.045510 (+2952359us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540004b854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.045601 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2551163648}
I20240426 13:19:44.262380 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.247946 24954 rpcz_store.cc:276] 0426 13:19:42.707286 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.707409 (+   123us) service_pool.cc:225] Handling call
0426 13:19:42.718403 (+ 10994us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.986584 (+268181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.986627 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:42.986682 (+    55us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:42.986722 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.045492 (+1058770us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.045535 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.045634 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616222464}
I20240426 13:19:44.269234 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1857) took 1041 ms. Trace:
W20240426 13:19:44.219581 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1843) took 2780 ms (client timeout 1948 ms). Trace:
W20240426 13:19:44.267935 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1759) took 2383 ms (client timeout 2960 ms). Trace:
I20240426 13:19:44.270128 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.270171 24948 rpcz_store.cc:269] 0426 13:19:41.884236 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.884481 (+   245us) service_pool.cc:225] Handling call
0426 13:19:44.035951 (+2151470us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:44.267874 (+231923us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2663680}
I20240426 13:19:44.059576 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.269606 25027 rpcz_store.cc:276] 0426 13:19:43.235990 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.433501 (+197511us) service_pool.cc:225] Handling call
0426 13:19:44.269175 (+835674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:44.245762 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 378) took 1116 ms. Trace:
I20240426 13:19:44.269253 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:44.271241 25023 rpcz_store.cc:269] 0426 13:19:41.439656 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.468744 (+ 29088us) service_pool.cc:225] Handling call
0426 13:19:44.219220 (+2750476us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.219466 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":711493376}
I20240426 13:19:44.273530 25102 rpcz_store.cc:276] 0426 13:19:43.129104 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.052981 (+923877us) service_pool.cc:225] Handling call
0426 13:19:44.245092 (+192111us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.245328 (+   236us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.245363 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.245419 (+    56us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.245465 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.245526 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.245653 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:44.267794 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.265866 24950 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:44.274739 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.061585 24955 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:44.108994 27263 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:44.286410 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.286692 27263 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:44.287292 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.223165 26912 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:44.224658 27303 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.271466 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.321283 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.216919 26969 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:44.276940 25104 rpcz_store.cc:276] 0426 13:19:42.762506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.847542 (+1085036us) service_pool.cc:225] Handling call
0426 13:19:43.858764 (+ 11222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.870466 (+ 11702us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.870513 (+    47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.870571 (+    58us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.870620 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.049587 (+178967us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.049642 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.049753 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26227584}
I20240426 13:19:44.279760 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.280128 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.325429 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.325704 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:44.285550 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.285710 27093 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.286105 26973 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240426 13:19:44.248350 26656 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:44.334302 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1361) took 3412 ms (client timeout 3135 ms). Trace:
I20240426 13:19:44.272688 27327 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.287864 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:44.314754 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:44.314841 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:44.316179 24921 connection.cc:232] RPC call timeout handler was delayed by -1.739396377999924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
I20240426 13:19:44.153260 27272 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:44.319108 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 303677us
W20240426 13:19:44.319707 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 299453us
I20240426 13:19:44.265726 27249 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:44.322665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.323614 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 289) took 1679 ms. Trace:
I20240426 13:19:44.321933 25025 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:44.325970 24954 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:44.061204 27203 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.326401 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.329124 25026 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:44.268684 27271 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.155087 27171 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.334239 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.060984 26839 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.328954 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.266247 27305 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.194319 27253 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.064123 27310 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.276476 27147 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.335186 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.263973 25098 rpcz_store.cc:276] 0426 13:19:42.753064 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.591117 (+838053us) service_pool.cc:225] Handling call
0426 13:19:43.615353 (+ 24236us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.635707 (+ 20354us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:43.840693 (+204986us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.976559 (+135866us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.106396 (+129837us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351
0426 13:19:44.107020 (+   624us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:44.222019 (+114999us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:19:44.222242 (+   223us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.222301 (+    59us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:44.222361 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.222444 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.222585 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16998,"prepare.queue_time_us":219565,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":252571,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":4067,"spinlock_wait_cycles":86125952,"thread_start_us":275872,"threads_started":2,"wal-append.queue_time_us":267292}
I20240426 13:19:44.642073 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.643128 25097 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:19:44.642447 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1530) took 2739 ms (client timeout 1811 ms). Trace:
I20240426 13:19:44.335918 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:44.644157 25101 rpcz_store.cc:269] 0426 13:19:41.902634 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.902828 (+   194us) service_pool.cc:225] Handling call
0426 13:19:44.043291 (+2140463us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:44.642387 (+599096us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15827200}
I20240426 13:19:44.649139 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.334764 25105 rpcz_store.cc:269] 0426 13:19:40.922112 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.923205 (+  1093us) service_pool.cc:225] Handling call
0426 13:19:44.334086 (+3410881us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.334211 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4171074816}
I20240426 13:19:44.389207 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.666317 24953 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:44.665113 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.389971 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.436969 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.663430 25022 rpcz_store.cc:276] 0426 13:19:42.644693 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.654289 (+  9596us) service_pool.cc:225] Handling call
0426 13:19:42.672933 (+ 18644us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.054455 (+381522us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.323390 (+1268935us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54000564d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.323560 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":338698112}
I20240426 13:19:44.645356 24947 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.668632 25101 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:44.437466 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.572603 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.669684 25619 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.644810 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.436576 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.674459 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.664669 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.675154 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:44.688746 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.689021 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.689479 25023 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:44.689709 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:44.625885 27344 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.298s	user 0.000s	sys 0.000s
I20240426 13:19:44.697934 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.387992 26656 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.436156 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.703068 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.703190 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.703321 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.704442 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.704700 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.704025 25101 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:44.728333 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.728961 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.705330 27272 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.731724 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:44.706091 27147 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.779304 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.324769 26969 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:44.706343 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:44.707670 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.662796 27327 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.662595 27271 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:44.704337 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.711114 26679 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.315701 26912 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:44.641340 27350 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:44.438690 27340 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
W20240426 13:19:44.708017 25622 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.727s	user 0.004s	sys 0.003s
I20240426 13:19:44.795338 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.708107 27310 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.795656 25622 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.710513 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.721212 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.704123 25102 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:44.798630 25619 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.799528 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.745836 27203 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.803014 25102 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:44.328874 27093 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.753847 25104 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:44.794785 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.021s	sys 0.088s
W20240426 13:19:44.804430 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.021s	sys 0.089s
W20240426 13:19:44.804823 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 530863us
W20240426 13:19:44.775146 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 672239us
I20240426 13:19:44.704279 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:44.710952 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:44.806063 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 95185us
I20240426 13:19:44.806560 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.776722 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.777537 25029 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:44.779871 24951 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.807886 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.808610 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.605222 26965 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.790613 27327 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.784262 24954 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.785681 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.819489 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.787294 25026 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:44.787381 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1487) took 2130 ms. Trace:
I20240426 13:19:44.795701 27147 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.796449 26679 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:44.724118 27346 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20240426 13:19:44.791024 25023 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:44.753017 27107 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.864s	user 0.009s	sys 0.000s
I20240426 13:19:44.323488 27113 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.791812 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.864027 24955 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:44.709884 27305 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.867169 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 383) took 1038 ms. Trace:
I20240426 13:19:44.813824 27339 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:44.794262 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.708225 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.796245 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.710774 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.797586 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.813828 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:44.783293 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.821588 26824 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:44.817235 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.788018 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.815797 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.828155 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.831229 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.788733 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57088.4us
I20240426 13:19:44.874223 25031 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:19:44.876689 24950 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.877048 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.872220 24953 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.722365 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:44.867525 25100 rpcz_store.cc:276] 0426 13:19:43.828926 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.445274 (+616348us) service_pool.cc:225] Handling call
0426 13:19:44.445614 (+   340us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.445820 (+   206us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:44.446081 (+   261us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.866855 (+420774us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.866904 (+    49us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.866960 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.867023 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.867113 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33845,"prepare.queue_time_us":153,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":35537,"thread_start_us":386594,"threads_started":1}
I20240426 13:19:44.809130 27093 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:44.751281 27323 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.689s	user 0.006s	sys 0.000s
W20240426 13:19:44.908288 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94770.7us
I20240426 13:19:44.910125 27353 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:44.907349 27327 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.910635 27353 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:44.911552 27353 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.912675 27327 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:44.914541 27273 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.646s	user 0.005s	sys 0.001s
I20240426 13:19:44.907469 27323 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.914964 27273 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.916067 24951 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:44.916863 27323 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.919133 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.880847 26969 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:44.919876 25097 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:44.920703 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.864053 27150 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:44.817862 26839 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.872182 25022 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:44.855028 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.820387 24956 rpcz_store.cc:276] 0426 13:19:42.650038 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.650176 (+   138us) service_pool.cc:225] Handling call
0426 13:19:42.664136 (+ 13960us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.671515 (+  7379us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:43.000235 (+328720us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.271567 (+271332us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.989189 (+717622us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b1800419aa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:19:43.989391 (+   202us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.989461 (+    70us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:43.989525 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.043916 (+ 54391us) spinlock_profiling.cc:243] Waited 8.43 ms on lock 0x7b1800419aa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:44.780321 (+736405us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.780366 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.780471 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31076,"prepare.queue_time_us":384786,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":737631,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":214,"spinlock_wait_cycles":45950976,"thread_start_us":565884,"threads_started":2,"wal-append.queue_time_us":415887,"wal-append.run_cpu_time_us":1312,"wal-append.run_wall_time_us":1012264}
I20240426 13:19:44.872400 25098 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:44.880148 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:44.869191 27113 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.882472 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.882864 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 374) took 1828 ms. Trace:
I20240426 13:19:44.883318 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.884157 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.929682 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54348.4us
I20240426 13:19:44.930119 25101 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:44.884990 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.885588 27263 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.886039 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.933877 25027 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.887025 24953 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:44.887480 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.935166 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:44.888127 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.889070 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 385) took 1021 ms. Trace:
I20240426 13:19:44.936328 25100 raft_consensus.cc:2354] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:44.890209 27253 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:44.890179 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 377) took 1766 ms. Trace:
I20240426 13:19:44.855878 26679 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.909734 27339 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:44.895335 26945 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.898056 27272 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.904578 25620 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.959841 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:44.827061 27147 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:44.868016 27340 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:19:44.921509 25615 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.909384 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:44.969954 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60645.9us
I20240426 13:19:44.924722 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.873003 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:44.931649 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 274839us
W20240426 13:19:44.931524 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4774963779999553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20240426 13:19:44.939376 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.964982 27147 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:44.802577 27350 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:44.936152 25105 rpcz_store.cc:276] 0426 13:19:43.867637 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.665408 (+797771us) service_pool.cc:225] Handling call
0426 13:19:44.706638 (+ 41230us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.706820 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:44.707064 (+   244us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.888741 (+181677us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.888798 (+    57us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:44.888841 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.888906 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.889012 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36154,"prepare.queue_time_us":3283,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":36919,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":145039,"threads_started":1}
I20240426 13:19:44.943387 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.946545 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.949271 27113 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.951097 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 338) took 1685 ms. Trace:
I20240426 13:19:44.951123 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.952445 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.971680 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:44.980692 25098 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:44.980800 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 179961us
I20240426 13:19:44.953061 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.957645 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.942104 26839 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.958945 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.869776 27107 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.983784 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.963198 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.962167 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.984968 25026 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:44.985028 25096 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:44.929350 25099 rpcz_store.cc:276] 0426 13:19:43.054780 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.943435 (+888655us) service_pool.cc:225] Handling call
0426 13:19:43.960442 (+ 17007us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.960761 (+   319us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.960794 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:43.961241 (+   447us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.027490 (+ 66249us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.027530 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.027596 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.882680 (+855084us) raft_consensus.cc:1671] finished
0426 13:19:44.882700 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.882813 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6001,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"thread_start_us":65928,"threads_started":1,"wal-append.queue_time_us":374326}
I20240426 13:19:44.941741 27323 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:44.973088 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4018963779999467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
I20240426 13:19:44.940363 25103 rpcz_store.cc:276] 0426 13:19:43.128726 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.011252 (+882526us) service_pool.cc:225] Handling call
0426 13:19:44.022085 (+ 10833us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:44.246104 (+224019us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.246135 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:44.265973 (+ 19838us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.439173 (+173200us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b180016d980. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:44.781773 (+342600us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:44.790665 (+  8892us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.796864 (+  6199us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.796937 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.889910 (+ 92973us) raft_consensus.cc:1671] finished
0426 13:19:44.890003 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.890124 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":290138,"prepare.queue_time_us":8614,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":245712,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":10196,"spinlock_wait_cycles":3844864,"thread_start_us":342393,"threads_started":1,"wal-append.queue_time_us":545294}
I20240426 13:19:44.985805 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.978000 25027 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.979938 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.980234 25022 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.987565 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.989001 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.989737 25022 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.990739 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.992398 25022 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:44.958276 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.979504 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.982865 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:44.994354 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:44.994957 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.965008 27114 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.995707 27114 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.995486 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.928951 27150 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.966279 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.997444 25101 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:44.965685 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.973202 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:44.999519 27150 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.999773 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.000095 27150 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.001160 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.002177 27141 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:44.995539 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.004776 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.006403 25101 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:44.986493 25031 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:45.008944 27349 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20240426 13:19:45.009308 27273 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.975172 27263 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.011271 27339 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.018411 27253 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.020301 27323 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.021600 27323 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.025918 27303 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.026564 27303 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.978195 26945 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.028574 26945 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.029021 26945 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.034564 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 288) took 2390 ms. Trace:
I20240426 13:19:45.035066 25028 rpcz_store.cc:276] 0426 13:19:42.644238 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.644414 (+   176us) service_pool.cc:225] Handling call
0426 13:19:44.220382 (+1575968us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.249241 (+ 28859us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:45.034278 (+785037us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54000b1150. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:45.034470 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3623674624}
I20240426 13:19:44.977442 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.040717 27331 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.041185 27331 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.044346 25022 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20240426 13:19:45.044692 26965 consensus_queue.cc:1159] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 1 last_received_current_leader { term: 2 index: 2 } }
I20240426 13:19:45.045964 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 305) took 1006 ms. Trace:
I20240426 13:19:45.046471 25022 rpcz_store.cc:276] 0426 13:19:44.039053 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.993398 (+954345us) service_pool.cc:225] Handling call
0426 13:19:44.993807 (+   409us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:45.044914 (+ 51107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:45.045667 (+   753us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:45.045892 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115507584}
I20240426 13:19:45.049144 25099 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:45.053315 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67993.7us
I20240426 13:19:45.060927 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.068588 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.073057 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.073998 25102 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:45.074604 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.074810 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.075186 25029 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.076005 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100406us
I20240426 13:19:45.077899 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.079581 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.080286 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:44.979306 24952 rpcz_store.cc:276] 0426 13:19:43.266067 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.544546 (+278479us) service_pool.cc:225] Handling call
0426 13:19:43.588208 (+ 43662us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:44.034033 (+445825us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.034065 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:44.034533 (+   468us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.294011 (+259478us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:44.294060 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.294119 (+    59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.950789 (+656670us) raft_consensus.cc:1671] finished
0426 13:19:44.950823 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.951024 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":445383,"raft.queue_time_us":447234,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":259038,"threads_started":1,"wal-append.queue_time_us":779292}
I20240426 13:19:45.082871 25105 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:45.083681 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100865us
I20240426 13:19:45.081315 26824 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.085165 26824 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.085522 27350 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.085582 26824 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.996170 27114 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.089442 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.098517 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.091374 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.103217 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.103989 25022 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.070590 27272 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.073091 27150 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.079146 27197 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.131412 27197 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.131811 27197 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:45.132196 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54819.8us
I20240426 13:19:45.133467 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.048801 26912 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.080778 27273 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.105448 27359 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.110854 27141 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.103278 27249 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.095185 27311 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.128314 27272 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.130918 27150 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.109095 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.101935 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:45.164547 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62731.1us
I20240426 13:19:45.165441 27359 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.165808 27359 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.166132 27359 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.166297 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.135301 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.167069 27113 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.142100 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.168668 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.169369 27113 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.169518 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.169793 27113 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.170157 27113 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.170271 27310 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.172477 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.154011 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.093464 27108 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.144320 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:45.097568 27107 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.173954 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:45.235303 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126327us
I20240426 13:19:45.143810 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:45.236460 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:45.240196 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67799.6us
W20240426 13:19:45.245815 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71905.9us
I20240426 13:19:45.282670 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.160498 26839 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.114931 26996 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240426 13:19:45.287326 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.283298 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.158403 27249 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.159303 26965 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20240426 13:19:45.286760 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.287120 25029 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.172549 27150 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.160440 27305 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.294332 25099 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.096324 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:45.295387 25098 raft_consensus.cc:595] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.288409 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.171967 27311 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.290768 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.238098 27350 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.160703 27273 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.185971 27339 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.245043 27310 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.296749 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.302222 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.283289 27107 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.174774 27365 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.306296 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.295156 26839 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.304968 27311 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.294203 27305 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.293206 27249 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.295307 26912 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:45.305291 27366 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.329186 27147 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.332067 27272 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.333122 27273 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.313375 27107 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:45.333945 27271 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.010s
I20240426 13:19:45.336827 27171 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.336611 27113 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.338644 27141 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.338235 27093 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:45.314245 26656 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.002s
I20240426 13:19:45.321117 27339 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.340008 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1390) took 2154 ms. Trace:
I20240426 13:19:45.340111 27310 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:45.377617 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.366034 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.333665 27365 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.289047 27370 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20240426 13:19:45.372918 27141 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.354373 27305 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.386719 27141 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.374439 25024 rpcz_store.cc:276] 0426 13:19:43.185710 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.188104 (+  2394us) service_pool.cc:225] Handling call
0426 13:19:43.250141 (+ 62037us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.263250 (+ 13109us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:43.548462 (+285212us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.618581 (+ 70119us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.618644 (+    63us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.618700 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.933375 (+1314675us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b180042b2c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:19:45.339786 (+406411us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:45.339838 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:45.339953 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":221623,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":112,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":67790464,"thread_start_us":339599,"threads_started":2,"wal-append.queue_time_us":61344,"wal-append.run_cpu_time_us":1076,"wal-append.run_wall_time_us":1005814}
W20240426 13:19:45.372242 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 135900us
I20240426 13:19:45.357723 27366 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.374388 25103 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.374763 25098 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.377810 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.331910 26945 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.401404 26945 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.363970 27107 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.401093 24854 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:45.404191 27366 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.404440 25100 raft_consensus.cc:2354] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:45.405783 27365 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.342810 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:45.380192 25620 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:45.408073 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.380360 25105 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.409354 24920 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
W20240426 13:19:45.367276 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 233793us
I20240426 13:19:45.354586 26839 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.399647 27113 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.395195 25102 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:45.409754 24920 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240426 13:19:45.410462 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.369285 27271 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.422514 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.423544 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:45.425189 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 137951us
W20240426 13:19:45.429807 27373 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:19:45.376214 26656 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:45.413177 27369 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20240426 13:19:45.432628 27379 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.433229 26656 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.389055 25617 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:45.421285 27171 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.436426 27365 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.436856 27365 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.437207 27365 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.426026 27370 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:45.440289 27370 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:45.441404 27370 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.442247 26839 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.443025 27370 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.445478 27147 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.449613 27197 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.451529 27197 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.453430 27171 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:45.453243 27150 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.454298 27303 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.381230 27310 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.456746 27378 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 13:19:45.458074 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 167001us
I20240426 13:19:45.459932 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:45.462949 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:45.470337 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46918.7us
I20240426 13:19:45.474308 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.412194 26965 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.396471 26824 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.480074 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.481266 24947 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:19:45.482758 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60355.1us
I20240426 13:19:45.494127 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.489166 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.482443 27093 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.497398 26656 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.497944 26656 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.498934 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.500222 27171 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.500658 27171 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.501029 27171 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.506039 26824 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.508404 25026 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:45.508922 27360 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.509469 27360 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.509675 26656 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.509946 27360 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.511086 27366 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.513065 27271 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.513916 25617 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:45.485302 26965 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.514957 26965 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.519402 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:45.520936 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:45.492264 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 195570us
I20240426 13:19:45.525949 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.498276 27379 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:45.527820 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.521718 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.565922 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.525903 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.569659 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.473403 26979 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20240426 13:19:45.573613 26824 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.496459 27253 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.574674 27150 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.615947 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.576932 27365 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.565428 27339 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20240426 13:19:45.578531 27107 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.579880 27171 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.581596 27310 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.582207 27263 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:45.526208 27176 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.585089 27311 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.631467 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.633224 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.634136 24949 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:45.588390 26679 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.635s	user 0.006s	sys 0.003s
I20240426 13:19:45.634900 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.635118 26679 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.636065 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:45.636663 27366 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.636852 24954 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:45.637149 26679 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.637686 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.638294 24956 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.515826 27276 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
W20240426 13:19:45.638533 27114 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.504s	user 0.007s	sys 0.000s
I20240426 13:19:45.638922 27114 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.589094 27303 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.640194 27311 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.566114 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.598702 25022 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20240426 13:19:45.641150 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75099.4us
I20240426 13:19:45.641927 25027 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:45.611903 27271 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.643239 25097 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.644017 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:45.625293 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:45.646600 24951 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.600126 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.493569 27197 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:45.600700 27174 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20240426 13:19:45.609406 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.604386 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1500) took 1272 ms. Trace:
W20240426 13:19:45.597251 27331 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.555s	user 0.006s	sys 0.006s
I20240426 13:19:45.649614 24947 rpcz_store.cc:276] 0426 13:19:44.331955 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.673203 (+341248us) service_pool.cc:225] Handling call
0426 13:19:45.604238 (+931035us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54003683d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:45.604335 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1550343040}
W20240426 13:19:45.654597 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75468.8us
I20240426 13:19:45.654735 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.655267 24948 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.575817 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97296.3us
I20240426 13:19:45.600944 27147 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.604516 27378 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:45.664059 27331 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.664459 27378 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.665652 24851 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 278) took 14168 ms (client timeout 15000 ms). Trace:
I20240426 13:19:45.665987 27331 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.667155 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.616739 27253 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.611845 26824 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.628610 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.676923 24955 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.678062 27174 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.678434 27174 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:45.678686 27360 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.679183 27174 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.630395 27176 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.680248 27176 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.682205 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:45.683477 27365 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.653143 27366 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20240426 13:19:45.616497 27108 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:45.680176 25029 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.676541 26824 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.621850 26945 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20240426 13:19:45.660804 27114 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.638418 25621 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:45.654208 27276 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.694223 27276 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.694586 27276 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.697618 27176 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.612309 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.711673 25025 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.653214 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.712509 24952 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.712836 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.713495 25621 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:45.683897 27359 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.517s	user 0.006s	sys 0.000s
I20240426 13:19:45.714380 27359 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:45.717000 24953 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.717885 27359 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.718376 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.718945 24953 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.718839 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.720686 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.721491 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.722002 25101 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.723982 27176 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:45.656066 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86460.1us
I20240426 13:19:45.648022 27249 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.733947 26912 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.660781 27107 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.663118 26969 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.738759 27249 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.667831 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.740845 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.741255 27360 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.666327 27378 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.671873 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:45.672513 27323 raft_consensus.cc:979] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:45.672597 25615 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:45.743372 27323 raft_consensus.cc:1067] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:45.673892 27171 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.673700 27141 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.675252 27375 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.745819 27375 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:45.746661 27375 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.747208 27141 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.667042 24854 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:45.676424 27253 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.700865 26824 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.666011 24851 rpcz_store.cc:269] 0426 13:19:31.497070 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.508165 (+ 11095us) service_pool.cc:225] Handling call
0426 13:19:45.665463 (+14157298us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:38.538554 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:38.539011 (+   457us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:38.539051 (+    40us) write_op.cc:427] Acquired schema lock
0426 13:19:38.539068 (+    17us) tablet.cc:660] Decoding operations
0426 13:19:38.552213 (+ 13145us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:38.552269 (+    56us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:38.552294 (+    25us) tablet.cc:683] Acquiring locks for 9 operations
0426 13:19:38.553020 (+   726us) tablet.cc:699] Row locks acquired
0426 13:19:38.553039 (+    19us) write_op.cc:260] PREPARE: finished
0426 13:19:38.553230 (+   191us) write_op.cc:270] Start()
0426 13:19:38.553348 (+   118us) write_op.cc:275] Timestamp: P: 1714137578553154 usec, L: 0
0426 13:19:38.553367 (+    19us) op_driver.cc:331] REPLICATION: starting
0426 13:19:38.554015 (+   648us) log.cc:842] Serialized 3639 byte log entry
0426 13:19:38.823009 (+268994us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8d3cee2e 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b
0426 13:19:39.262119 (+439110us) op_driver.cc:444] REPLICATION: finished
0426 13:19:39.542809 (+280690us) write_op.cc:300] APPLY: starting
0426 13:19:39.542896 (+    87us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:39.543105 (+   209us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:39.543129 (+    24us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:39.544093 (+   964us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:39.544295 (+   202us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0426 13:19:39.544340 (+    45us) write_op.cc:311] APPLY: finished
0426 13:19:39.544613 (+   273us) log.cc:842] Serialized 71 byte log entry
0426 13:19:39.544758 (+   145us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:39.544878 (+   120us) write_op.cc:446] Released schema lock
0426 13:19:39.544918 (+    40us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:39.545193 (+   275us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:39.545221 (+    28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":179346,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":249717,"apply.run_cpu_time_us":2741,"apply.run_wall_time_us":2742,"num_ops":9,"prepare.queue_time_us":296811,"prepare.run_cpu_time_us":8471,"prepare.run_wall_time_us":302064,"raft.queue_time_us":274958,"raft.run_cpu_time_us":15685,"raft.run_wall_time_us":196958,"replication_time_us":663363,"spinlock_wait_cycles":3329408,"thread_start_us":693353,"threads_started":4,"wal-append.queue_time_us":313078,"wal-append.run_cpu_time_us":26946,"wal-append.run_wall_time_us":1810187}]]}
I20240426 13:19:45.753557 26824 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.702781 24947 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.703881 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.704890 27174 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.706422 27147 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.724887 26965 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.763911 26965 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.725555 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.765759 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.765933 24953 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240426 13:19:45.726780 27114 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.728727 25023 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.771271 26945 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:45.772965 26945 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:45.773329 26945 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.776672 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.732142 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.743947 26713 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.734095 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.735260 25620 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:45.657619 27386 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 13:19:45.735270 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78543.6us
I20240426 13:19:45.737080 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:45.784448 27147 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.784750 24996 leader_election.cc:398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:45.785068 24996 leader_election.cc:401] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:45.785251 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51232.8us
I20240426 13:19:45.747979 27375 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.785992 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:45.613561 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 129090us
I20240426 13:19:45.738364 24948 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:45.741369 24949 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.671284 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.673559 27305 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.605114 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 83495.2us
I20240426 13:19:45.761744 27380 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:45.722153 24952 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.783113 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55088.4us
I20240426 13:19:45.791853 24951 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:45.783689 26713 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.796506 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.793771 27386 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:45.803717 27176 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.804181 27305 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.797279 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.797255 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:45.804900 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.798079 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.805501 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:45.794618 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.806154 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.795357 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.806564 25026 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.774230 26945 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.807655 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.829972 24949 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.810213 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.803061 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:45.811432 27305 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:45.811950 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.812484 27171 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.813218 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.806773 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:45.793459 27147 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.793141 27323 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.793365 27114 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.823377 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.823763 25589 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:45.824141 25104 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:45.824285 27029 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.825174 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.825172 27108 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:45.816496 27141 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.832531 24956 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.834718 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.837241 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.808696 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.775241 27276 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.839495 25025 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.846434 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:45.881464 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57892.8us
W20240426 13:19:45.886929 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80829.7us
I20240426 13:19:45.923949 25098 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:45.924433 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:45.887216 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92663.9us
W20240426 13:19:45.931489 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96896.4us
I20240426 13:19:45.931738 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.851974 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.932252 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.810680 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:45.925396 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.933281 25097 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.926541 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.881655 25096 raft_consensus.cc:2362] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5. Re-sending same reply.
I20240426 13:19:45.934310 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.935075 24955 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20240426 13:19:45.936638 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.937078 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.937326 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:45.927337 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90191.9us
I20240426 13:19:45.886539 25100 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.928903 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.950138 25096 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:45.938509 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.951438 25096 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:45.808203 26713 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.888116 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.938470 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.967375 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.967873 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.968814 27382 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:45.969555 26996 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:45.947327 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.970448 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.970580 27271 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.970654 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.971089 27271 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.971297 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.971540 27271 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.936724 26912 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.972074 26912 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.972456 26912 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.972738 27323 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:45.972895 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.973320 27323 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.973798 27377 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:45.974576 27372 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.974254 27323 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.975055 27372 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.976141 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.946621 26656 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:19:45.979585 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.979719 27393 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.980365 27350 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:45.946092 27113 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:45.980700 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:45.981120 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.955729 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.955848 27385 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:45.975884 27372 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.982738 27197 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.986102 27203 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.986459 27203 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.987226 27203 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.956542 27108 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:45.958508 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.988365 27203 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.974287 27377 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.988756 27376 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:45.989348 27376 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.989449 27377 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.990281 27376 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.959631 27176 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.952980 26965 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20240426 13:19:45.992656 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.988448 27108 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.993435 25026 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.993527 27360 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:45.994154 27360 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:45.994277 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.995411 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.995764 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:45.960527 27394 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20240426 13:19:45.996810 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.996666 27253 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.963119 27276 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.998121 25025 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.995972 25026 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.998534 27253 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.999133 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.000713 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.951778 27311 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:45.838743 27305 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.969326 27352 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:45.979151 26839 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.935865 27114 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.971554 27249 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.994010 27108 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.995056 27360 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.002123 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.015089 25030 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.885905 27386 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.932615 25099 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:45.961042 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:46.004431 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.000799 27107 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.005287 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.988451 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.953966 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.008666 25028 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:46.009092 25103 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.935708 26824 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.948184 27378 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20240426 13:19:46.010027 27311 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.948014 26945 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.017535 24956 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:45.936298 27310 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:46.021245 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.052104 27378 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:46.052326 27263 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:46.052597 27378 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.052896 27263 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.021839 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:46.025462 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.963764 27150 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:46.029224 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.960312 27280 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:46.037556 26965 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:46.025506 25026 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:46.011682 27393 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.012023 27305 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:46.038338 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84442.7us
I20240426 13:19:46.048352 26839 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:46.041381 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52989.9us
I20240426 13:19:46.140022 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.014513 27249 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.838934 27359 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.031226 25030 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:46.049201 27331 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.078985 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.000468 25621 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:46.080315 27276 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.138446 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:46.033788 27386 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:46.069630 27263 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.034550 27353 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240426 13:19:46.152760 26656 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.070475 26965 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:46.153024 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74142.9us
I20240426 13:19:46.049440 27113 raft_consensus.cc:2741] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240426 13:19:46.159421 27366 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.080312 27150 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.054646 26969 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:46.053627 27378 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.040695 27147 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.151926 27311 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.180404 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.139328 27107 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:19:46.150619 24951 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:46.206516 24951 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.144397 27249 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.190855 25022 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:46.160498 27382 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.200142 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:46.191807 26965 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:46.199995 26824 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:46.190250 27398 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20240426 13:19:46.206079 25098 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.153285 27391 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20240426 13:19:46.191545 27350 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.150455 27310 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20240426 13:19:46.184298 27365 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:46.220158 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.184942 27271 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.230726 25026 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.223090 25029 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:46.210189 27263 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.209932 27114 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.233700 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:46.219159 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:46.234421 25098 raft_consensus.cc:595] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.237489 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.219518 27359 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.220762 27377 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.228121 27365 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:46.254544 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:46.251183 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109127us
W20240426 13:19:46.263170 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72458.9us
I20240426 13:19:46.233705 27339 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:46.274632 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.293742 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.253705 27391 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:46.295588 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.305019 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:46.276687 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.260537 27107 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:46.321599 27399 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20240426 13:19:46.322185 27401 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20240426 13:19:46.322343 26996 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.324595 27372 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:46.325479 27029 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:46.325990 27366 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.326853 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:46.408581 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 81809.7us
I20240426 13:19:46.327526 27400 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.328258 27404 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.269629 27365 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:46.330250 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53667.7us
I20240426 13:19:46.332726 27107 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:46.384428 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 133514us
W20240426 13:19:46.387367 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 150596us
I20240426 13:19:46.266314 27359 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.307081 27197 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:46.251645 27114 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.292672 27339 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.232378 27331 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.304023 27391 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.252933 27405 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20240426 13:19:46.334072 27276 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.430075 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.438964 24921 leader_election.cc:398] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:46.400646 27372 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.399878 27029 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.271307 27377 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.451700 24921 leader_election.cc:401] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:46.269177 27150 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.424976 27360 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.442616 27176 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20240426 13:19:46.435113 27400 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.448467 27174 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.436342 27366 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:46.428318 27386 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.468864 27107 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.477814 26713 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.000s
I20240426 13:19:46.478765 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:46.400915 27350 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.460975 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.307833 26824 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.410995 26996 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.489454 25620 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:46.462388 27197 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.527873 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.486188 27171 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:46.499042 27366 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:46.533941 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103979us
I20240426 13:19:46.538924 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:46.541626 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132322us
I20240426 13:19:46.467968 26912 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.550489 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.531648 27310 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:46.491086 27305 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.561724 27310 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.576370 27310 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.546792 27171 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.580338 27400 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.581418 27414 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20240426 13:19:46.581192 27420 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20240426 13:19:46.537927 27379 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.533890 27271 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.540993 27366 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.498893 27174 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.533704 27386 raft_consensus.cc:2741] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240426 13:19:46.561866 26839 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240426 13:19:46.584973 25026 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:46.642203 24955 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:46.570062 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.571532 24951 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.533027 27360 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.527781 26713 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.583128 27108 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:46.562625 27107 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.643743 24956 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:46.572294 27402 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
I20240426 13:19:46.583348 27253 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:46.654093 25024 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
W20240426 13:19:46.658000 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106536us
I20240426 13:19:46.580371 27411 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:46.676717 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.641420 27276 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.682469 25102 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.683275 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.683336 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.685086 25103 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:46.643776 27409 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.536015 27391 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.689899 27114 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:46.690660 27391 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.692999 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.693770 25100 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.694233 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.595758 27303 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20240426 13:19:46.557822 27359 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.656081 27385 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.700592 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.700286 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.661429 27386 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.601934 26965 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.704908 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44823.7us
I20240426 13:19:46.558284 27029 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.583176 27372 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.709255 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.738226 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.738978 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.739145 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.739768 25101 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.739730 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:46.742632 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:46.743616 24949 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20240426 13:19:46.744649 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.745524 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.745746 25101 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:46.745981 27174 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.746315 25022 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.747092 25101 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:46.741844 27403 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20240426 13:19:46.702587 27409 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.750064 26656 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:46.750623 27409 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:46.748870 26713 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:46.753356 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.753360 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.756320 25030 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.754030 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.756623 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.754881 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.757210 25022 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.703236 27108 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.763092 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.760564 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.688304 27276 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:46.766782 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.767371 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:46.768057 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.740451 27377 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:46.767781 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102933us
I20240426 13:19:46.774390 25029 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:46.759665 27407 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.308s	user 0.001s	sys 0.001s
I20240426 13:19:46.758123 26839 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.708190 27386 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.692221 27428 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.208s
I20240426 13:19:46.763468 27271 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:46.801355 26965 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.705538 27385 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.804329 27435 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.840893 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.785548 26656 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:46.808445 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 312) took 1003 ms. Trace:
I20240426 13:19:46.845223 25024 rpcz_store.cc:276] 0426 13:19:45.805262 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.805394 (+   132us) service_pool.cc:225] Handling call
0426 13:19:46.808184 (+1002790us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:46.808368 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1767372544}
I20240426 13:19:46.677875 27197 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.809859 26945 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.755s	user 0.003s	sys 0.006s
I20240426 13:19:46.736286 27114 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240426 13:19:46.812971 27310 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.814848 27305 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:46.816416 27276 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:19:46.818485 27425 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20240426 13:19:46.818598 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:46.819260 27423 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
W20240426 13:19:46.819653 27263 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.571s	user 0.008s	sys 0.001s
I20240426 13:19:46.822573 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.776021 27439 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.823267 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.823355 27419 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.222s
I20240426 13:19:46.684410 27411 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.815979 24951 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.775346 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:46.834971 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 265045us
I20240426 13:19:46.805657 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.797916 27377 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:46.799273 26996 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.797299 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:46.828955 26912 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.862954 25022 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:46.863174 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102646us
I20240426 13:19:46.866156 25098 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:19:46.839643 26839 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.968575 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.945035 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 1813) took 1137 ms. Trace:
I20240426 13:19:46.970256 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.905211 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.842613 27271 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.966324 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.755725 27360 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.910809 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 168256us
I20240426 13:19:46.909351 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.954155 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.972941 24949 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:46.913283 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.847745 27435 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.932099 27422 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.903846 25589 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:19:46.951797 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.018246 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.971088 25022 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:46.975641 24947 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:46.926832 27310 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.020081 25589 leader_election.cc:401] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:46.977272 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:46.978555 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.980965 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:47.023783 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45324.4us
I20240426 13:19:47.025468 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.027469 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.028355 24955 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.968919 27377 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:46.969971 24951 rpcz_store.cc:276] 0426 13:19:45.805941 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.806669 (+   728us) service_pool.cc:225] Handling call
0426 13:19:46.814991 (+1008322us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9030ef37 00007fda9525bb50 00007fda8e949754 00007fda8e9496e2 00007fda8e949611 00007fda8e94958c 00007fda8e948842 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:46.934530 (+119539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724330880}
I20240426 13:19:46.963351 27386 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:46.942282 27415 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
W20240426 13:19:46.943158 27436 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20240426 13:19:47.020689 25030 raft_consensus.cc:2354] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:46.947659 27311 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.022802 25024 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:46.961555 27385 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:46.923660 27249 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.703s	user 0.009s	sys 0.000s
I20240426 13:19:46.975777 27271 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:46.952214 27411 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:47.037707 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.976334 27439 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:47.040560 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63405us
I20240426 13:19:47.039499 27377 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.042814 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:47.043318 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.017424 26656 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:46.906213 27108 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.052410 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.981783 25617 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:47.063724 27445 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:47.065423 27435 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:47.065851 26969 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.862s	user 0.004s	sys 0.005s
I20240426 13:19:46.984400 26713 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20240426 13:19:47.066943 27447 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:47.069334 27365 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.625s	user 0.005s	sys 0.005s
W20240426 13:19:47.037498 27423 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.069s	user 0.004s	sys 0.000s
I20240426 13:19:47.037132 27276 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:47.074986 27171 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:47.040138 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 136476us
I20240426 13:19:47.088064 24955 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:47.043265 27422 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:47.099431 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56199.2us
I20240426 13:19:47.101967 25024 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240426 13:19:47.104508 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66919.1us
I20240426 13:19:47.126163 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.117163 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:47.126971 25104 raft_consensus.cc:595] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:47.112342 27359 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:47.113368 27386 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:47.117702 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.121824 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.127305 25104 raft_consensus.cc:603] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:47.118664 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:47.128445 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:47.100508 27439 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.131471 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 377) took 1076 ms. Trace:
I20240426 13:19:47.134493 24952 rpcz_store.cc:276] 0426 13:19:46.055359 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.055654 (+   295us) service_pool.cc:225] Handling call
0426 13:19:46.056064 (+   410us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.056560 (+   496us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:46.056603 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.080131 (+ 23528us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.266900 (+186769us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:46.266944 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.267015 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.131269 (+864254us) raft_consensus.cc:1671] finished
0426 13:19:47.131297 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.131417 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":17580,"thread_start_us":79367,"threads_started":1,"wal-append.queue_time_us":491513}
I20240426 13:19:47.115916 27259 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:47.115581 26969 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.115983 27447 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:47.077143 27249 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.137707 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.146929 24955 raft_consensus.cc:2354] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:47.141356 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.138126 25031 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.139252 24921 leader_election.cc:398] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:47.184747 24921 leader_election.cc:401] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:47.119451 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:47.138135 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:47.185267 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46173.2us
W20240426 13:19:47.185657 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66276.8us
I20240426 13:19:47.188313 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.188903 25102 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.189188 25102 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:47.189188 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.117354 27365 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.143412 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.143455 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.138962 27113 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:47.144510 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.144623 24952 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:47.122926 27171 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.129223 27276 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:47.232522 25624 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:47.159406 24947 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20240426 13:19:47.172995 24949 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:47.142799 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.190115 25615 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.190546 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.141894 27400 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:47.235002 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.194188 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.235606 26824 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:47.235034 27451 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20240426 13:19:47.236305 27412 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.235641 27400 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.236836 27412 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.216722 27310 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:47.186820 27409 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:47.229534 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.237426 27400 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:47.252007 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.252663 25623 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.235195 26713 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:47.219288 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 319) took 1057 ms. Trace:
I20240426 13:19:47.194737 27113 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:47.219836 27449 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:47.240016 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.235811 25024 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.147294 27447 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.247367 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1571) took 1083 ms. Trace:
I20240426 13:19:47.278474 24949 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:47.279202 24947 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:47.282296 24955 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:47.246903 27310 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:47.237573 27171 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.256621 25617 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240426 13:19:47.462451 27276 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:47.470973 27382 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:47.443267 27456 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20240426 13:19:47.471514 27276 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.473466 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.474273 25029 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.474428 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.475566 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.475910 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.476343 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:47.477092 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.477500 27310 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:47.477803 24956 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:47.478206 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.478529 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.478914 24952 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:47.480285 27412 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:47.481094 27366 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.839s	user 0.011s	sys 0.000s
I20240426 13:19:47.481619 27276 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:47.481469 27366 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.482784 24952 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:47.483402 27366 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:47.483808 24952 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:47.486204 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.487187 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.441879 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.488298 25027 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:47.442437 27360 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20240426 13:19:47.489818 27360 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:47.490156 27360 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:47.367566 27259 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:19:47.326668 27386 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20240426 13:19:47.468675 27171 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.275610 25099 rpcz_store.cc:276] 0426 13:19:46.163933 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.164248 (+   315us) service_pool.cc:225] Handling call
0426 13:19:46.164635 (+   387us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.190015 (+ 25380us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:46.206223 (+ 16208us) log.cc:842] Serialized 14 byte log entry
0426 13:19:46.313829 (+107606us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b180037f6c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:19:46.453388 (+139559us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.465514 (+ 12126us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.465665 (+   151us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:46.465712 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.465802 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.196021 (+730219us) raft_consensus.cc:1671] finished
0426 13:19:47.247176 (+ 51155us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.247311 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60848,"prepare.queue_time_us":16200,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":49980,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":17211,"spinlock_wait_cycles":4250880,"thread_start_us":124568,"threads_started":1,"wal-append.queue_time_us":333569}
I20240426 13:19:47.499595 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.328370 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.501613 25024 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:47.386093 25023 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:47.294009 27439 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.470005 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.297230 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:47.282649 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.503402 25624 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:47.467248 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 278133us
I20240426 13:19:47.505594 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.493949 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.506906 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.507037 24954 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.258136 25104 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:47.514029 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:47.514669 25104 raft_consensus.cc:595] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.522223 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:47.284412 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 156779us
I20240426 13:19:47.284170 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.571482 24947 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:47.255259 25028 rpcz_store.cc:276] 0426 13:19:46.161452 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.170502 (+  9050us) service_pool.cc:225] Handling call
0426 13:19:46.187093 (+ 16591us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.424204 (+237111us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:46.424250 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.451999 (+ 27749us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.640585 (+188586us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:46.662069 (+ 21484us) log.cc:842] Serialized 14 byte log entry
0426 13:19:46.662277 (+   208us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.662346 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.219007 (+556661us) raft_consensus.cc:1671] finished
0426 13:19:47.219034 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.219216 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25178,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4201,"raft.queue_time_us":138396,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":6379,"thread_start_us":375738,"threads_started":2,"wal-append.queue_time_us":287791}
I20240426 13:19:47.586484 24952 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:47.247311 26839 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:47.588943 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66809.6us
W20240426 13:19:47.600641 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107889us
I20240426 13:19:47.256935 26824 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:47.442492 27446 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.270s	user 0.000s	sys 0.000s
W20240426 13:19:47.444218 27454 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20240426 13:19:47.274245 27385 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:47.275380 27113 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:47.428926 27448 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20240426 13:19:47.471544 27382 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.605326 25104 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.603878 24995 leader_election.cc:398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:47.606719 25096 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:47.292286 27350 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.611050 25024 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:47.423394 26679 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.414131 27422 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.595892 27249 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:47.274549 27271 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.587005 27409 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.596966 27445 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:47.640084 25024 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.640504 25024 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:47.641571 25096 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20240426 13:19:47.281409 27449 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:47.602432 27400 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:47.602198 27440 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.622009 25622 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:47.642313 25096 pending_rounds.cc:76] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:19:47.642105 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:47.641410 25024 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.654855 24953 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:47.479914 27447 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.635123 27365 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.604586 26839 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.650171 25096 pending_rounds.cc:97] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240426 13:19:47.651985 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 13:19:47.644742 27377 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.001s
I20240426 13:19:47.669526 25096 raft_consensus.cc:2866] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.645987 27449 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.689270 25623 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.631036 27385 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:47.689841 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92167.1us
I20240426 13:19:47.634394 26679 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.721588 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:47.613832 26824 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.646521 27391 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 13:19:47.633078 27350 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.721478 24956 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.636958 27422 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.732986 27462 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20240426 13:19:47.754952 25101 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:47.761634 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 158254us
I20240426 13:19:47.658075 27416 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
W20240426 13:19:47.730650 27359 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.587s	user 0.005s	sys 0.000s
I20240426 13:19:47.791667 27359 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.658526 27445 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20240426 13:19:47.796226 27339 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.804316 25097 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:47.754256 27271 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:47.771492 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 160190us
I20240426 13:19:47.658849 27440 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.762862 25099 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:47.733557 27107 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.065s	user 0.012s	sys 0.000s
I20240426 13:19:47.778699 25616 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:47.638602 27360 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.782931 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.823246 25096 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.688143 27377 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.817915 27400 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240426 13:19:47.732815 24954 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:47.815757 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 182110us
W20240426 13:19:47.816381 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 176939us
W20240426 13:19:47.824082 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101989us
I20240426 13:19:47.823004 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.849267 25026 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:19:47.849653 25026 pending_rounds.cc:76] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:47.849880 25026 pending_rounds.cc:97] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:19:47.850104 25026 raft_consensus.cc:2866] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.851837 26679 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.853405 26656 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.752s	user 0.010s	sys 0.000s
I20240426 13:19:47.853683 26824 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.860496 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 67928.3us
I20240426 13:19:47.843394 27327 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.853833 27479 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:47.879109 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49846.6us
I20240426 13:19:47.843909 27409 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.847719 27271 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.853849 26656 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:47.918846 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:47.919852 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:47.913522 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83078us
I20240426 13:19:47.926323 25104 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240426 13:19:47.926498 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63896.9us
I20240426 13:19:47.880941 27310 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.934515 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.932317 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.886157 27327 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.912649 27476 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.932622 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.945520 25103 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240426 13:19:47.945936 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1404) took 1270 ms. Trace:
I20240426 13:19:47.946218 25103 rpcz_store.cc:276] 0426 13:19:46.675709 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.924864 (+1249155us) service_pool.cc:225] Handling call
0426 13:19:47.945885 (+ 21021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:47.950054 27376 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.950767 27376 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.954334 25026 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:47.900388 27409 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.939827 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1572) took 1488 ms (client timeout 1570 ms). Trace:
I20240426 13:19:47.941566 25104 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240426 13:19:47.958786 25104 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:19:47.959570 27485 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.930706 26839 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.904101 27271 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.930967 27447 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:47.960776 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1817) took 1957 ms (client timeout 2207 ms). Trace:
I20240426 13:19:47.961863 25026 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.972618 25026 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:47.972366 24950 rpcz_store.cc:269] 0426 13:19:46.002900 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.003377 (+   477us) service_pool.cc:225] Handling call
0426 13:19:47.960519 (+1957142us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.960685 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2440853888}
I20240426 13:19:47.963764 27485 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.959187 25104 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
W20240426 13:19:47.958573 25098 rpcz_store.cc:269] 0426 13:19:46.451076 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.451235 (+   159us) service_pool.cc:225] Handling call
0426 13:19:47.939580 (+1488345us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.939748 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":774660992}
I20240426 13:19:47.979717 25104 raft_consensus.cc:2894] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20240426 13:19:47.983242 25104 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.967506 27410 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.894114 27479 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.993016 27487 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:47.995551 27487 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.997233 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1901) took 1991 ms (client timeout 2004 ms). Trace:
W20240426 13:19:47.997691 25023 rpcz_store.cc:269] 0426 13:19:46.006174 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.006330 (+   156us) service_pool.cc:225] Handling call
0426 13:19:47.997070 (+1990740us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000b0754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.997179 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2524216704}
I20240426 13:19:47.999644 25029 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:47.959945 27382 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.987874 25026 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.975071 27385 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.997267 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1416) took 1297 ms (client timeout 1644 ms). Trace:
W20240426 13:19:48.004228 25027 rpcz_store.cc:269] 0426 13:19:46.700222 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.700363 (+   141us) service_pool.cc:225] Handling call
0426 13:19:47.997111 (+1296748us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.997222 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1809761024}
I20240426 13:19:48.006111 27422 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.006196 27482 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20240426 13:19:48.006937 27271 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.009966 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.981786 27350 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.996773 27477 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:19:47.991868 26839 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.992106 27113 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.014655 27377 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.177999 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1517) took 1017 ms. Trace:
I20240426 13:19:48.003087 26824 raft_consensus.cc:979] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
W20240426 13:19:48.188834 27484 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20240426 13:19:48.006939 27485 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.016817 25026 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.098532 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.130702 25096 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:48.188632 24955 rpcz_store.cc:276] 0426 13:19:47.160175 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.160341 (+   166us) service_pool.cc:225] Handling call
0426 13:19:48.177774 (+1017433us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54003683d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.177932 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106162560}
I20240426 13:19:48.275151 27491 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20240426 13:19:47.961301 27476 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.278656 27476 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.059891 27350 raft_consensus.cc:2741] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20240426 13:19:48.280508 27479 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.047384 27493 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.267s
I20240426 13:19:48.281155 27271 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:48.279945 25026 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:48.282241 27271 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.282699 27271 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.284790 27310 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.286626 27376 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:48.287678 27311 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.233s	user 0.011s	sys 0.000s
I20240426 13:19:48.288292 25100 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:48.288791 27409 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.290192 27449 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.578s	user 0.007s	sys 0.001s
I20240426 13:19:48.290643 27449 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:48.292189 27460 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.283s	user 0.001s	sys 0.001s
I20240426 13:19:48.293977 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1916) took 1183 ms. Trace:
I20240426 13:19:48.294361 25022 rpcz_store.cc:276] 0426 13:19:47.110094 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.111613 (+  1519us) service_pool.cc:225] Handling call
0426 13:19:47.111944 (+   331us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.120869 (+  8925us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.120916 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.125753 (+  4837us) log.cc:842] Serialized 26 byte log entry
0426 13:19:47.327134 (+201381us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:47.669014 (+341880us) log.cc:842] Serialized 14 byte log entry
0426 13:19:47.680567 (+ 11553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:47.680665 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:48.293712 (+613047us) raft_consensus.cc:1671] finished
0426 13:19:48.293742 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:48.293909 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15064,"prepare.queue_time_us":453050,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":22395,"raft.queue_time_us":190033,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":11696,"spinlock_wait_cycles":6908800,"thread_start_us":441710,"threads_started":2,"wal-append.queue_time_us":540903}
W20240426 13:19:48.297905 27171 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.701s	user 0.009s	sys 0.000s
I20240426 13:19:48.299319 27391 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.300122 27391 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:48.300496 27391 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.301447 27391 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.186738 27486 raft_consensus.cc:979] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:48.176371 27339 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:48.306677 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:48.174964 27327 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.319289 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.326148 24947 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.011s	sys 0.033s
W20240426 13:19:48.328500 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.012s	sys 0.033s
I20240426 13:19:48.328696 25104 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:48.332407 24948 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:48.302991 27391 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:48.102155 27495 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:48.104168 27440 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.284827 27487 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.365794 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.009s	sys 0.027s
W20240426 13:19:48.391413 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.009s	sys 0.027s
I20240426 13:19:48.438408 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.102850 27461 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.346s
I20240426 13:19:48.440261 25622 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:48.437733 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.018s	sys 0.033s
W20240426 13:19:48.461055 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.041s	sys 0.047s
I20240426 13:19:48.007946 27410 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:48.462754 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.019s	sys 0.034s
W20240426 13:19:48.547757 24981 heartbeater.cc:644] Failed to heartbeat to 127.23.61.62:34153 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.61.62:34153 timed out after 15.000s (SENT)
W20240426 13:19:48.466403 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.035s	sys 0.039s
I20240426 13:19:48.546418 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.546195 27406 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.553886 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1565) took 2701 ms. Trace:
W20240426 13:19:48.462821 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.041s	sys 0.047s
I20240426 13:19:48.561342 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.595407 25097 rpcz_store.cc:276] 0426 13:19:45.852133 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.948867 (+ 96734us) service_pool.cc:225] Handling call
0426 13:19:46.012474 (+ 63607us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.804808 (+1792334us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.553561 (+748753us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.553815 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4120238336}
I20240426 13:19:48.339776 27377 raft_consensus.cc:1067] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
W20240426 13:19:48.346223 27416 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.026s	sys 0.021s
I20240426 13:19:48.583815 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:48.586963 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.047s	sys 0.060s
W20240426 13:19:48.606496 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 735238us
I20240426 13:19:48.534396 27492 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.574860 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.346189 27350 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.437994 27432 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:48.466481 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.043s	sys 0.043s
I20240426 13:19:48.101998 27490 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.432s
I20240426 13:19:48.340699 27495 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:48.603642 27429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.001s	sys 0.016s
I20240426 13:19:48.437355 27489 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.547060 27474 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:48.368782 27376 raft_consensus.cc:1067] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:48.605439 24950 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:48.587109 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.035s	sys 0.073s
W20240426 13:19:48.626830 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.035s	sys 0.073s
I20240426 13:19:48.607501 24956 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:48.627123 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 690362us
I20240426 13:19:48.608583 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:48.547266 27481 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.566737 27378 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:48.548611 27113 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.610428 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:48.576900 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:48.631635 25620 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.631875 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.606513 27499 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.334311 26839 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.508370 27487 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.624342 24921 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:48.625058 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.609668 26945 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.604346 25616 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
W20240426 13:19:48.612622 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.044s	sys 0.052s
I20240426 13:19:48.641731 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:48.666569 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.05685e+06us
I20240426 13:19:48.667083 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.614037 27479 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:48.631623 27429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.001s	sys 0.016s
I20240426 13:19:48.628758 27311 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.616233 27406 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.618152 27432 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.664232 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.626297 27271 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.619411 27449 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.668017 27498 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.617735 27350 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:48.673830 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.053s	sys 0.153s
I20240426 13:19:48.676793 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:48.679570 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.053s	sys 0.154s
I20240426 13:19:48.680285 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.675348 27487 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.686909 27487 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.653282 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.689383 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.714509 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:48.715787 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.659336 27474 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.689638 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.690665 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.691237 25023 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:48.723824 25027 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.672012 27113 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.652627 27504 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.723618 25026 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:48.678481 27376 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.733466 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:48.714359 27490 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.671391 26945 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.747225 26945 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.748871 27029 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:48.749423 27432 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.753253 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1559) took 2921 ms. Trace:
I20240426 13:19:48.753733 25105 rpcz_store.cc:276] 0426 13:19:45.831641 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.831771 (+   130us) service_pool.cc:225] Handling call
0426 13:19:45.925070 (+ 93299us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.133906 (+1208836us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.752935 (+1619029us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.753184 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2751953792}
I20240426 13:19:48.729579 25621 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:48.724613 27271 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:48.757548 26839 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.654312 27499 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.758692 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1581) took 1266 ms. Trace:
I20240426 13:19:48.759222 25099 rpcz_store.cc:276] 0426 13:19:47.492286 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.499327 (+  7041us) service_pool.cc:225] Handling call
0426 13:19:48.758474 (+1259147us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.758629 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":603227392}
I20240426 13:19:48.669052 27377 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.626039 27492 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.720500 27491 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:48.635039 27489 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:48.737238 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.041s	sys 0.136s
W20240426 13:19:48.778059 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.041s	sys 0.137s
I20240426 13:19:48.761621 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:48.778594 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 853118us
I20240426 13:19:48.779960 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.725945 27449 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.726168 27496 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.678432 27311 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.781374 27449 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.781754 27496 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.782888 27311 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.784276 27496 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.781376 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102272us
I20240426 13:19:48.734622 27504 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.676324 27479 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.787997 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.758500 27497 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.790992 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.797441 25022 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:48.668777 27481 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.799821 27481 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.801452 25028 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:48.801821 25028 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.802461 25028 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.803544 25028 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:48.798174 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 569) took 2779 ms. Trace:
I20240426 13:19:48.804816 25615 rpcz_store.cc:276] 0426 13:19:46.010789 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.010950 (+   161us) service_pool.cc:225] Handling call
0426 13:19:46.025088 (+ 14138us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.234343 (+1209255us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.790184 (+1555841us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.790333 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2679076736}
I20240426 13:19:48.744840 27480 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20240426 13:19:48.808821 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 379) took 1516 ms. Trace:
I20240426 13:19:48.809160 25623 rpcz_store.cc:276] 0426 13:19:47.292486 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.292608 (+   122us) service_pool.cc:225] Handling call
0426 13:19:47.445977 (+153369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.690521 (+244544us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.808616 (+1118095us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540058e654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.808774 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558836480}
I20240426 13:19:48.735946 27171 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:48.797998 24954 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:48.811390 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1531) took 1368 ms (client timeout 1588 ms). Trace:
W20240426 13:19:48.814795 24948 rpcz_store.cc:269] 0426 13:19:47.442984 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.475362 (+ 32378us) service_pool.cc:225] Handling call
0426 13:19:48.811222 (+1335860us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.811331 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1958315776}
I20240426 13:19:48.796244 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:48.733568 27113 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.821743 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:48.824472 25022 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:48.859385 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:48.790522 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.013s	sys 0.011s
I20240426 13:19:48.816632 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:48.844931 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48781.4us
W20240426 13:19:48.865545 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.018s	sys 0.011s
I20240426 13:19:48.846285 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.866792 25027 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:48.672861 27495 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.783859 27449 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.790458 26839 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:48.737896 27376 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.900252 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.725620 27498 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.908366 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.908902 25027 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.910110 27498 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.904786 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.905468 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46184.2us
I20240426 13:19:48.905851 25022 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:19:48.910835 25022 pending_rounds.cc:76] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:48.911067 25022 pending_rounds.cc:97] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:19:48.911067 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.911652 25022 raft_consensus.cc:2866] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:19:48.913229 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 341884us
I20240426 13:19:48.913913 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:48.914197 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97663.7us
I20240426 13:19:48.916303 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.917551 25623 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:48.917974 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.918547 25103 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:48.918929 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.920621 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.920846 25027 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:48.769944 27490 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.922885 27461 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.923710 27490 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.923530 27461 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.924527 27490 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.925464 27510 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:48.869316 27378 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.930167 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.930768 25097 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:48.876613 27391 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.878643 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.927742 25022 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:48.915597 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 248465us
I20240426 13:19:48.876734 27323 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:48.928632 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 237984us
I20240426 13:19:48.917490 27498 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:48.932813 27502 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.933291 27500 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:48.779873 27491 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.912515 26713 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20240426 13:19:48.929441 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.926110 27461 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.906535 27481 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.906755 26839 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.777290 27377 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.930191 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:48.943852 25022 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:48.944766 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.946580 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.555s (SENT) [suppressed 11 similar messages]
I20240426 13:19:48.966765 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.928123 27487 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.931712 27150 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:48.951627 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.838204 27410 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.950847 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.871023 27480 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20240426 13:19:48.967536 25022 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.943351 27323 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.970336 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.987895 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.988404 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.952492 27508 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.976384 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.994451 25098 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:48.995548 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.979439 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.982476 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.789498 27497 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.999061 27497 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.912837 27433 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20240426 13:19:48.966487 26839 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:48.999743 27433 consensus_queue.cc:1225] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:48.958729 27504 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.965914 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:48.967730 27500 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.989252 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:49.014976 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:49.015329 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 99094.2us
I20240426 13:19:48.978399 27406 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.969058 27350 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.011528 27492 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.025617 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.871377 27474 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.020334 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.031080 25097 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:49.032183 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.994311 27323 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.968554 26945 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240426 13:19:48.987530 27461 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.016177 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.026454 25026 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:49.027122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 106579us
I20240426 13:19:49.008642 27496 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.968886 27481 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:49.027303 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80749.4us
I20240426 13:19:48.993093 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:49.043663 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50681.2us
I20240426 13:19:49.009869 27150 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:49.039731 27461 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.051146 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.016410 27500 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:49.013222 27513 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.990329 27515 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:49.058892 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:49.061787 25025 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:49.009452 27517 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:49.054538 27527 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:49.051791 26973 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:49.062567 26969 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.062939 26973 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:49.063117 27527 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:49.063413 26973 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:49.063071 26969 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.055195 27386 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.972680 27502 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.012425 27514 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.068905 27386 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.069072 27514 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.070369 27502 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.071310 27386 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.071391 27514 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.049448 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.080847 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.081530 25097 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:49.084347 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.088856 27489 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.057580 27479 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.093979 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.094120 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:49.094854 27489 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.096519 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.097123 25097 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:49.095247 27489 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.099452 27489 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:49.099512 27491 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.099769 27481 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.098031 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:49.085225 27480 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:49.058499 26713 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:49.109566 25026 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:49.050361 27496 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.087167 27113 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:49.110817 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51996us
I20240426 13:19:49.060084 27406 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:49.129385 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.130554 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.131278 25026 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:49.130941 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:49.132539 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.136749 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.098114 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.093223 27376 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:49.056703 27350 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20240426 13:19:49.120213 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:49.112252 27499 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.113870 27479 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:49.153441 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.123728 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.125303 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:49.143353 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 128705us
I20240426 13:19:49.146347 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:49.147889 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.168889 25097 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:49.148623 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.174412 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.122812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.117045 27447 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.162240 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.177700 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.178725 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.178897 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.171144 25026 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:49.165868 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.180830 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.183930 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.186244 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:49.190666 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.182559 27029 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:49.192220 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.192793 25023 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:49.193665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:49.194778 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:49.194871 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.195493 25023 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:49.195742 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.196311 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.010154 27508 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.196921 25023 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:49.110532 27481 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20240426 13:19:49.195979 27511 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20240426 13:19:49.215600 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.135049 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.199123 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.216545 25026 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:49.151007 27504 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.151796 27491 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.206861 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:49.208230 24948 pending_rounds.cc:76] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:49.209460 25623 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240426 13:19:49.155493 27377 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.149991 27113 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.234694 27375 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.234588 27113 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.214164 27108 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.094888 27150 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.218796 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:49.221582 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:49.213471 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.214354 27029 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:49.169948 27400 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:49.226724 24948 pending_rounds.cc:97] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:49.179695 27447 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.231899 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.196511 27472 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.189564 27439 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:49.230485 25623 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 3
I20240426 13:19:49.232821 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.305907 25623 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20240426 13:19:49.306361 25623 raft_consensus.cc:2894] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
W20240426 13:19:49.161151 27339 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.828s	user 0.008s	sys 0.002s
I20240426 13:19:49.232386 25022 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:49.233870 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 172012us
I20240426 13:19:49.206058 27461 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.197542 27479 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:49.197044 27376 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.214517 27249 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:49.235109 27375 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.234606 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.245110 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.248973 25026 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:49.248481 25028 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:49.251120 24948 raft_consensus.cc:2866] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:49.251315 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:49.251621 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.311297 27518 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:49.228530 27504 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.244666 27108 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.233538 27491 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.311564 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:49.724879 27489 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.625s	user 0.015s	sys 0.002s
I20240426 13:19:49.766384 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.758545 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.771018 27489 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.772163 27108 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:49.772063 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.772634 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.773651 25105 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:49.722714 24947 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:49.253546 27447 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:49.713008 27498 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.755s	user 0.009s	sys 0.000s
I20240426 13:19:49.775422 27498 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:49.773620 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.807754 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:49.806277 25099 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:49.779803 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1434) took 2588 ms. Trace:
I20240426 13:19:49.725091 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.813813 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:49.334086 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.325438 25623 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:49.773639 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.817117 25028 rpcz_store.cc:276] 0426 13:19:47.191572 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.574197 (+382625us) service_pool.cc:225] Handling call
0426 13:19:48.555686 (+981489us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:48.555981 (+   295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:48.801364 (+245383us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000f30a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:19:49.779554 (+978190us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:49.779746 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279718912}
I20240426 13:19:49.779007 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:49.722890 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 325580us
I20240426 13:19:49.818636 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.819509 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.327428 25101 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240426 13:19:49.762609 25615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.000s	sys 0.010s
W20240426 13:19:49.326956 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70938.5us
W20240426 13:19:49.725585 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 398485us
I20240426 13:19:49.820966 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.329931 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1585) took 1023 ms. Trace:
I20240426 13:19:49.822434 25096 rpcz_store.cc:276] 0426 13:19:48.306439 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.306955 (+   516us) service_pool.cc:225] Handling call
0426 13:19:48.307377 (+   422us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:48.342930 (+ 35553us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:48.342970 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:48.343524 (+   554us) log.cc:842] Serialized 26 byte log entry
0426 13:19:48.683245 (+339721us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:48.683279 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.683341 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:49.329673 (+646332us) raft_consensus.cc:1671] finished
0426 13:19:49.329699 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:49.329862 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158248,"raft.queue_time_us":38608,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":58630,"thread_start_us":339402,"threads_started":1,"wal-append.queue_time_us":443155}
W20240426 13:19:49.824769 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.000s	sys 0.014s
W20240426 13:19:49.555997 27323 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.001s
I20240426 13:19:49.252925 27497 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:49.487362 27506 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.770s	user 0.005s	sys 0.000s
W20240426 13:19:49.709823 26839 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.710s	user 0.011s	sys 0.000s
I20240426 13:19:49.400072 27524 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:49.810868 27276 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.822017 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:49.399390 27521 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20240426 13:19:49.820129 27331 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:49.825065 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.000s	sys 0.014s
I20240426 13:19:49.833947 26945 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.255275 27400 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.850106 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.733376 27483 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:49.814797 27505 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:19:49.672065 27536 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.780355 27249 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.812196 27375 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:49.327833 25102 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.431254 27439 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.857791 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.602424 27508 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.859613 25105 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:49.820986 25615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.000s	sys 0.011s
I20240426 13:19:49.887236 27475 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:49.671411 27540 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.775127 27479 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.326934 27514 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.547s
I20240426 13:19:49.368206 27502 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.493s
W20240426 13:19:49.882094 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60134.4us
I20240426 13:19:50.003721 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.807264 27518 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:49.883683 27524 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.884222 25618 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:49.884503 27276 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.862715 26839 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.885836 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1921) took 2641 ms. Trace:
I20240426 13:19:49.823608 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:49.355244 27433 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.508s
I20240426 13:19:49.355443 27515 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.526s
W20240426 13:19:49.897169 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 645587us
I20240426 13:19:49.856976 27497 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.897652 25028 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:49.459384 26973 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.815594 27376 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.918640 25023 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:50.013494 27518 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:49.828668 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.002s	sys 0.011s
W20240426 13:19:50.048138 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.002s	sys 0.011s
W20240426 13:19:49.944360 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.028s	sys 0.057s
I20240426 13:19:49.535872 27029 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.312721 27472 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.835987 27498 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:49.893683 27400 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:49.945991 27439 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.917075 27249 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:49.837095 27323 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.016374 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.920117 27536 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.018803 25030 rpcz_store.cc:276] 0426 13:19:47.244695 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.719432 (+474737us) service_pool.cc:225] Handling call
0426 13:19:47.719755 (+   323us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.720117 (+   362us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.720145 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.720619 (+   474us) log.cc:842] Serialized 26 byte log entry
0426 13:19:48.391749 (+671130us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:48.391790 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.391876 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:49.151483 (+759607us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:49.885557 (+734074us) raft_consensus.cc:1671] finished
0426 13:19:49.885577 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:49.885780 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19321,"raft.queue_time_us":192729,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":14549,"spinlock_wait_cycles":336450432,"thread_start_us":670761,"threads_started":1,"wal-append.queue_time_us":891365}
W20240426 13:19:50.004518 27500 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.930s	user 0.014s	sys 0.000s
I20240426 13:19:49.897634 26945 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.022733 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:50.021893 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 198389us
I20240426 13:19:50.026484 25026 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
W20240426 13:19:50.018558 27511 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:19:49.860553 27461 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:50.019032 27475 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:50.048887 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.028s	sys 0.057s
I20240426 13:19:50.019258 27524 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.021440 27502 consensus_queue.cc:1225] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:19:49.903172 27113 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.052695 25623 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.054078 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.124426 25103 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:50.124538 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1450) took 1113 ms. Trace:
I20240426 13:19:50.125233 25023 rpcz_store.cc:276] 0426 13:19:49.011442 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.046290 (+1034848us) service_pool.cc:225] Handling call
0426 13:19:50.124466 (+ 78176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:49.903517 27483 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.063464 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1831) took 3089 ms. Trace:
I20240426 13:19:50.126860 24953 rpcz_store.cc:276] 0426 13:19:46.974402 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.976177 (+  1775us) service_pool.cc:225] Handling call
0426 13:19:46.976495 (+   318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.655319 (+678824us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.063160 (+2407841us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54000a69d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.063392 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1559599872}
I20240426 13:19:50.000779 27540 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:50.066494 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50212.6us
I20240426 13:19:50.133322 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.067027 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:50.117960 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 268828us
I20240426 13:19:50.128391 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.022148 27474 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 13:19:50.052590 27504 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.137090 24953 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:50.136965 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:50.138738 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 330) took 1132 ms. Trace:
I20240426 13:19:50.139251 25030 rpcz_store.cc:276] 0426 13:19:49.006404 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.066690 (+1060286us) service_pool.cc:225] Handling call
0426 13:19:50.138690 (+ 72000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.140338 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:50.140689 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.147918 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.149495 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.236411 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 332) took 1227 ms. Trace:
I20240426 13:19:50.237005 25030 rpcz_store.cc:276] 0426 13:19:49.008714 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.140400 (+1131686us) service_pool.cc:225] Handling call
0426 13:19:50.236323 (+ 95923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.127900 27491 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.134953 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.250615 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.141244 27276 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.145574 27113 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:50.095321 27500 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:50.271644 27543 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.364s	user 0.001s	sys 0.000s
I20240426 13:19:50.144430 27376 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.272307 27380 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:50.021360 27554 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20240426 13:19:50.263881 27433 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:50.268460 27491 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.103668 27536 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.279006 25023 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:50.284322 27323 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.261685 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:50.272526 27113 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:50.280876 27380 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:50.290894 27380 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.277529 27551 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:50.285833 25616 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:50.292287 27113 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.296911 25030 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:50.297611 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1448) took 1286 ms. Trace:
I20240426 13:19:50.297983 25030 rpcz_store.cc:276] 0426 13:19:49.010887 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.261343 (+1250456us) service_pool.cc:225] Handling call
0426 13:19:50.297541 (+ 36198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.283022 27433 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:50.318240 27545 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20240426 13:19:50.303223 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:50.323282 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 329) took 1317 ms. Trace:
W20240426 13:19:50.323545 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1576) took 3359 ms (client timeout 1602 ms). Trace:
I20240426 13:19:50.323734 25030 rpcz_store.cc:276] 0426 13:19:49.006195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.302247 (+1296052us) service_pool.cc:225] Handling call
0426 13:19:50.323208 (+ 20961us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.324288 25100 rpcz_store.cc:269] 0426 13:19:46.964148 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.965948 (+  1800us) service_pool.cc:225] Handling call
0426 13:19:50.323323 (+3357375us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.323458 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3027776256}
I20240426 13:19:50.294597 26973 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.326100 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 540) took 1929 ms (client timeout 2116 ms). Trace:
I20240426 13:19:50.326267 27555 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:50.326539 25624 rpcz_store.cc:269] 0426 13:19:48.396301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.396432 (+   131us) service_pool.cc:225] Handling call
0426 13:19:50.325927 (+1929495us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.326022 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28568832}
I20240426 13:19:50.327831 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.326134 27556 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20240426 13:19:50.321732 27433 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.288741 27491 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.316183 27497 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.321470 27323 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:50.342499 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.362090 27375 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.364447 25618 raft_consensus.cc:2354] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:19:50.365787 27500 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.370179 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1430) took 3179 ms. Trace:
I20240426 13:19:50.370568 25029 rpcz_store.cc:276] 0426 13:19:47.191008 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.474724 (+283716us) service_pool.cc:225] Handling call
0426 13:19:47.999358 (+524634us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:48.005393 (+  6035us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.369989 (+2364596us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54000b0750. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.370122 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1205726080}
I20240426 13:19:50.371834 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.372864 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:50.373492 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1445) took 1389 ms. Trace:
I20240426 13:19:50.373852 25029 rpcz_store.cc:276] 0426 13:19:48.984504 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.371569 (+1387065us) service_pool.cc:225] Handling call
0426 13:19:50.373428 (+  1859us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.374847 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.336846 27439 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.375779 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1447) took 1374 ms. Trace:
I20240426 13:19:50.381063 25029 rpcz_store.cc:276] 0426 13:19:49.001861 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.374600 (+1372739us) service_pool.cc:225] Handling call
0426 13:19:50.375718 (+  1118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.327072 27555 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:50.363876 27249 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.382411 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.385246 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 334) took 1365 ms. Trace:
I20240426 13:19:50.385797 25029 rpcz_store.cc:276] 0426 13:19:49.019732 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.382118 (+1362386us) service_pool.cc:225] Handling call
0426 13:19:50.385181 (+  3063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.358752 27497 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:50.340246 27540 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.388355 27249 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.387858 27552 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20240426 13:19:50.378877 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.396178 27447 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.621s	user 0.002s	sys 0.008s
I20240426 13:19:50.396656 27447 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.397938 25621 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:50.409684 27483 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.411538 27549 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.396s	user 0.000s	sys 0.000s
I20240426 13:19:50.411487 27553 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:50.415498 27375 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.418197 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1948) took 1200 ms. Trace:
I20240426 13:19:50.418675 25023 rpcz_store.cc:276] 0426 13:19:49.217458 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.126215 (+908757us) service_pool.cc:225] Handling call
0426 13:19:50.417977 (+291762us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54004155d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.418124 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333289728}
I20240426 13:19:50.424674 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.393741 27491 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.425721 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 542) took 1925 ms (client timeout 1754 ms). Trace:
W20240426 13:19:50.434533 25622 rpcz_store.cc:269] 0426 13:19:48.500605 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.500803 (+   198us) service_pool.cc:225] Handling call
0426 13:19:50.424481 (+1923678us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:50.425639 (+  1158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29923712}
I20240426 13:19:50.435787 25023 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:50.436712 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1438) took 1502 ms. Trace:
I20240426 13:19:50.437196 25023 rpcz_store.cc:276] 0426 13:19:48.934158 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.419686 (+1485528us) service_pool.cc:225] Handling call
0426 13:19:50.426712 (+  7026us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.436397 (+  9685us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.436653 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.333374 27400 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.442950 25023 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:50.325196 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.446480 27366 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:50.478374 27366 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:50.446552 27550 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20240426 13:19:50.410760 26839 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.448894 27555 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.449465 27447 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.449509 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1944) took 1233 ms. Trace:
I20240426 13:19:50.509565 25028 rpcz_store.cc:276] 0426 13:19:49.216493 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.818422 (+601929us) service_pool.cc:225] Handling call
0426 13:19:50.449319 (+630897us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.449452 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179757440}
I20240426 13:19:50.453027 26945 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.453794 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 398) took 2859 ms. Trace:
I20240426 13:19:50.513690 24952 rpcz_store.cc:276] 0426 13:19:47.594372 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.598566 (+  4194us) service_pool.cc:225] Handling call
0426 13:19:47.598875 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.647148 (+ 48273us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.647179 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:47.647230 (+    51us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:47.647299 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.453553 (+2806254us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.453607 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.453734 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110426880}
I20240426 13:19:50.455552 27391 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.458902 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 424) took 1508 ms. Trace:
I20240426 13:19:50.466346 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:50.466374 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1440) took 1520 ms. Trace:
I20240426 13:19:50.515468 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1454) took 1357 ms. Trace:
I20240426 13:19:50.477819 27400 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.515267 25098 rpcz_store.cc:276] 0426 13:19:48.950194 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.996570 (+ 46376us) service_pool.cc:225] Handling call
0426 13:19:48.996863 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.003577 (+  6714us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:49.049908 (+ 46331us) log.cc:842] Serialized 14 byte log entry
0426 13:19:49.219068 (+169160us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:49.219165 (+    97us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:49.219208 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.458613 (+1239405us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.458672 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.458833 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39857,"prepare.queue_time_us":51503,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":36686,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7348,"spinlock_wait_cycles":14805376,"thread_start_us":164842,"threads_started":2,"wal-append.queue_time_us":558632}
I20240426 13:19:50.412446 27553 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.479348 27366 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.506666 27360 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:50.431450 27113 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.515467 25023 rpcz_store.cc:276] 0426 13:19:48.946316 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.438561 (+1492245us) service_pool.cc:225] Handling call
0426 13:19:50.442697 (+  4136us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.466058 (+ 23361us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.466310 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.516171 25030 rpcz_store.cc:276] 0426 13:19:49.158611 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.324918 (+1166307us) service_pool.cc:225] Handling call
0426 13:19:50.515414 (+190496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.523582 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1441) took 1573 ms. Trace:
W20240426 13:19:50.516952 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.019s	sys 0.038s
I20240426 13:19:50.460201 27433 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:50.525645 25028 rpcz_store.cc:276] 0426 13:19:48.950401 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.510616 (+1560215us) service_pool.cc:225] Handling call
0426 13:19:50.522933 (+ 12317us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.523166 (+   233us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.523220 (+    54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.523277 (+    57us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.523320 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.523386 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.523518 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.527766 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.019s	sys 0.038s
I20240426 13:19:50.525871 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 323) took 1527 ms. Trace:
I20240426 13:19:50.528266 25030 rpcz_store.cc:276] 0426 13:19:48.998543 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.525160 (+1526617us) service_pool.cc:225] Handling call
0426 13:19:50.525429 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.525579 (+   150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.525602 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.525651 (+    49us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:50.525683 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.525735 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.525827 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.527427 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 325) took 1528 ms. Trace:
I20240426 13:19:50.530772 25028 rpcz_store.cc:276] 0426 13:19:48.999003 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.526674 (+1527671us) service_pool.cc:225] Handling call
0426 13:19:50.526990 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.527144 (+   154us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.527168 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.527211 (+    43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:50.527241 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.527289 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.527381 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11648}
I20240426 13:19:50.527268 27366 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.533807 27372 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.534086 27113 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.453954 27524 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.522414 27391 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:50.551348 25024 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:50.542940 27518 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.570276 25028 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:50.575201 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1449) took 1564 ms. Trace:
I20240426 13:19:50.575693 25028 rpcz_store.cc:276] 0426 13:19:49.011191 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.531805 (+1520614us) service_pool.cc:225] Handling call
0426 13:19:50.569962 (+ 38157us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.574951 (+  4989us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.575149 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.530253 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 327) took 1530 ms. Trace:
W20240426 13:19:50.577278 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 543) took 1455 ms (client timeout 1802 ms). Trace:
I20240426 13:19:50.579119 25030 rpcz_store.cc:276] 0426 13:19:48.999472 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.529406 (+1529934us) service_pool.cc:225] Handling call
0426 13:19:50.529708 (+   302us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.529892 (+   184us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.529927 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.529978 (+    51us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:50.530020 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.530081 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.530198 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.580471 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 423) took 1632 ms. Trace:
I20240426 13:19:50.581570 25029 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:19:50.583665 25103 rpcz_store.cc:276] 0426 13:19:48.947990 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.951372 (+  3382us) service_pool.cc:225] Handling call
0426 13:19:50.580255 (+1628883us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.580408 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2641634560}
I20240426 13:19:50.581766 25028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:50.586066 25028 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
W20240426 13:19:50.582139 24948 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.019s	sys 0.090s
I20240426 13:19:50.583002 25023 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:50.586704 25028 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:50.585306 25030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:50.589660 25030 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:50.590096 25030 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.589542 24948 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.019s	sys 0.090s
I20240426 13:19:50.592101 24951 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.592856 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1536) took 3147 ms. Trace:
I20240426 13:19:50.593251 24951 rpcz_store.cc:276] 0426 13:19:47.445017 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.488578 (+ 43561us) service_pool.cc:225] Handling call
0426 13:19:50.591639 (+3103061us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:50.591874 (+   235us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.592584 (+   710us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.592795 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7502720}
I20240426 13:19:50.594426 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1853) took 1388 ms. Trace:
I20240426 13:19:50.591300 27113 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:50.602798 27113 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.603555 27524 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.604691 27113 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.602813 27507 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:50.606150 27507 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:50.606843 27507 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.608706 27507 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.579942 25621 rpcz_store.cc:269] 0426 13:19:49.122211 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.128157 (+  5946us) service_pool.cc:225] Handling call
0426 13:19:50.550595 (+1422438us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.574482 (+ 23887us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2708740864}
I20240426 13:19:50.587884 27433 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.615916 27489 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.598909 24948 rpcz_store.cc:276] 0426 13:19:49.206034 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.207485 (+  1451us) service_pool.cc:225] Handling call
0426 13:19:49.207952 (+   467us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.827203 (+619251us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.827235 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.827754 (+   519us) log.cc:842] Serialized 52 byte log entry
0426 13:19:50.003203 (+175449us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.592012 (+588809us) log.cc:842] Serialized 14 byte log entry
0426 13:19:50.594050 (+  2038us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.594138 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:50.594192 (+    54us) raft_consensus.cc:1671] finished
0426 13:19:50.594210 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.594354 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3725,"prepare.queue_time_us":18889,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":131831,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":57335,"thread_start_us":680332,"threads_started":2,"wal-append.queue_time_us":420419}
I20240426 13:19:50.631072 27391 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.621198 27553 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.648443 27518 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.655200 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.040s	sys 0.053s
W20240426 13:19:50.655718 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.041s	sys 0.054s
W20240426 13:19:50.656075 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 757730us
W20240426 13:19:50.660133 24954 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.021s	sys 0.027s
W20240426 13:19:50.660602 24954 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.021s	sys 0.027s
I20240426 13:19:50.672601 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1850) took 1503 ms. Trace:
I20240426 13:19:50.673135 24954 rpcz_store.cc:276] 0426 13:19:49.169448 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.170867 (+  1419us) service_pool.cc:225] Handling call
0426 13:19:49.216615 (+ 45748us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.233081 (+ 16466us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.233122 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.233637 (+   515us) log.cc:842] Serialized 52 byte log entry
0426 13:19:50.048568 (+814931us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:50.058442 (+  9874us) log.cc:842] Serialized 14 byte log entry
0426 13:19:50.409566 (+351124us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:50.448165 (+ 38599us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b1800330880. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:50.672203 (+224038us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.672281 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:50.672343 (+    62us) raft_consensus.cc:1671] finished
0426 13:19:50.672362 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.672537 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6726,"prepare.queue_time_us":19343,"prepare.run_cpu_time_us":288,"prepare.run_wall_time_us":5418,"raft.queue_time_us":94232,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":7897,"spinlock_wait_cycles":7952896,"thread_start_us":1348579,"threads_started":2,"wal-append.queue_time_us":651941}
I20240426 13:19:50.618814 27380 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.681494 27380 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.671715 27422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.027s	sys 0.070s
I20240426 13:19:50.687474 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.671607 27518 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.690016 27518 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.690408 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.686499 27422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.027s	sys 0.070s
I20240426 13:19:50.693284 25027 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:50.691879 27518 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.694288 27465 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.695098 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1435) took 1779 ms. Trace:
I20240426 13:19:50.695454 25027 rpcz_store.cc:276] 0426 13:19:48.915279 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.922202 (+  6923us) service_pool.cc:225] Handling call
0426 13:19:48.922536 (+   334us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.693686 (+1771150us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.694878 (+  1192us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.695046 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4071686144}
I20240426 13:19:50.697870 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1942) took 1481 ms. Trace:
I20240426 13:19:50.698931 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.698346 27490 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.702296 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.704260 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.706056 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.694896 27465 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.700588 25027 rpcz_store.cc:276] 0426 13:19:49.215969 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.697037 (+1481068us) service_pool.cc:225] Handling call
0426 13:19:50.697341 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.697519 (+   178us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.697552 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.697613 (+    61us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.697653 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.697712 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.697815 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.705701 25030 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:50.718724 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.004s	sys 0.018s
W20240426 13:19:50.723551 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.004s	sys 0.018s
W20240426 13:19:50.724015 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 662822us
I20240426 13:19:50.730345 25028 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.731587 25027 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.706248 25616 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:50.755687 25030 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.758700 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.759333 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:50.759600 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.725780 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:50.760491 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.761524 24948 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:19:50.758010 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:50.708187 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.757455 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.006s	sys 0.034s
I20240426 13:19:51.014122 24947 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
W20240426 13:19:51.014735 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.010s	sys 0.037s
W20240426 13:19:51.015129 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 992412us
I20240426 13:19:50.787461 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51366 (request call id 1941) took 1546 ms. Trace:
I20240426 13:19:51.016024 25030 rpcz_store.cc:276] 0426 13:19:49.215726 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.583700 (+1367974us) service_pool.cc:225] Handling call
0426 13:19:50.762491 (+178791us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.762799 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.031s	sys 0.014s
W20240426 13:19:51.018327 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.033s	sys 0.019s
W20240426 13:19:50.983039 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.017s	sys 0.015s
W20240426 13:19:51.020416 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.018s	sys 0.016s
W20240426 13:19:51.021075 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 871630us
I20240426 13:19:51.021967 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.825256 25028 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.023545 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.024626 24953 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:51.025629 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.026851 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.023545 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:50.840837 25615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.009s	sys 0.040s
W20240426 13:19:51.034616 25615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.097s	user 0.009s	sys 0.056s
I20240426 13:19:50.764607 24952 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:51.035215 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:50.770659 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.007s	sys 0.028s
W20240426 13:19:51.038030 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.019s	sys 0.047s
I20240426 13:19:51.038690 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51366 (request call id 1940) took 1807 ms. Trace:
I20240426 13:19:50.831346 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:51.039534 25028 rpcz_store.cc:276] 0426 13:19:49.215453 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.577020 (+1361567us) service_pool.cc:225] Handling call
0426 13:19:51.023287 (+446267us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:51.041091 24955 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.010s	sys 0.016s
I20240426 13:19:51.041298 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 868) took 1896 ms. Trace:
W20240426 13:19:51.039677 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 899382us
W20240426 13:19:51.032855 27029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.006s	sys 0.020s
W20240426 13:19:50.862509 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.001s	sys 0.018s
W20240426 13:19:51.043365 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.005s	sys 0.021s
I20240426 13:19:51.041671 25030 rpcz_store.cc:276] 0426 13:19:49.144848 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.039033 (+1894185us) service_pool.cc:225] Handling call
0426 13:19:51.041257 (+  2224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:51.045105 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046356 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046784 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046962 25028 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.047351 25030 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.862757 25097 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.027s	sys 0.042s
I20240426 13:19:51.050101 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.041440 24955 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.010s	sys 0.016s
I20240426 13:19:51.048889 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.049113 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:51.038877 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 313146us
I20240426 13:19:50.829102 27508 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.841413 27564 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:50.911764 27486 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.174s	user 0.008s	sys 0.037s
W20240426 13:19:50.862684 27515 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.009s	sys 0.010s
W20240426 13:19:51.050733 25097 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.147s	user 0.029s	sys 0.048s
I20240426 13:19:51.052232 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.055955 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.761978 27553 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.067451 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.040954 27197 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.017s	user 0.015s	sys 0.028s
I20240426 13:19:51.061890 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.065797 27486 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.331s	user 0.015s	sys 0.038s
I20240426 13:19:50.788638 27360 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:50.788185 27420 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:50.755434 27372 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.296s
W20240426 13:19:51.043233 27029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.006s	sys 0.020s
I20240426 13:19:51.058821 25096 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:51.094818 25103 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.074616 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1594) took 1186 ms. Trace:
I20240426 13:19:50.719275 27490 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.096536 27564 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.050964 27479 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:51.094749 25105 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.116650 27564 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.117959 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1430) took 2164 ms (client timeout 2180 ms). Trace:
I20240426 13:19:51.118325 27271 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:51.118880 27271 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:51.057550 27429 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.120122 27429 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.073880 27515 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.023s	user 0.011s	sys 0.015s
I20240426 13:19:51.068495 27366 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:51.098758 27197 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.016s	sys 0.028s
I20240426 13:19:51.062822 27524 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.098356 27553 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.113252 27565 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20240426 13:19:51.063246 27435 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.124426 25620 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.078431 26824 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20240426 13:19:51.106897 27360 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:51.119853 27271 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.066982 27508 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.151098 27271 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.130911 27366 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.154474 27366 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.132565 27524 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.155928 27524 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.161401 27508 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.161965 27503 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.428s	user 0.009s	sys 0.016s
I20240426 13:19:51.162392 27553 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.163520 27490 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.164180 27429 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.164533 27465 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.165594 27470 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:51.166126 27470 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:51.166658 27507 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.557s	user 0.008s	sys 0.000s
I20240426 13:19:51.167119 27470 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.170640 27496 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.178622 27498 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:51.179018 27498 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:51.179414 27494 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.527s	user 0.007s	sys 0.017s
I20240426 13:19:51.179801 27498 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.181061 27498 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:50.790335 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.225684 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:51.226495 25024 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.016s	sys 0.006s
W20240426 13:19:51.227151 25024 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.016s	sys 0.006s
W20240426 13:19:51.229324 24995 connection.cc:232] RPC call timeout handler was delayed by -1.5798693399999593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20240426 13:19:51.051586 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1855) took 1819 ms. Trace:
W20240426 13:19:51.229985 24995 connection.cc:232] RPC call timeout handler was delayed by -1.575369339999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20240426 13:19:51.230477 24995 connection.cc:232] RPC call timeout handler was delayed by -1.5627693400000453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20240426 13:19:51.066381 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.096308 25098 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.125628 25105 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:51.118360 25104 rpcz_store.cc:269] 0426 13:19:48.953832 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.964411 (+ 10579us) service_pool.cc:225] Handling call
0426 13:19:51.117772 (+2153361us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.117900 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2117535488}
W20240426 13:19:51.231215 24995 connection.cc:232] RPC call timeout handler was delayed by -1.2135693399999354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
I20240426 13:19:51.245966 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.230528 24955 rpcz_store.cc:276] 0426 13:19:49.232295 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.232481 (+   186us) service_pool.cc:225] Handling call
0426 13:19:49.234580 (+  2099us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.234932 (+   352us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.234970 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.244199 (+  9229us) log.cc:842] Serialized 52 byte log entry
0426 13:19:49.886749 (+642550us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:49.917607 (+ 30858us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.051253 (+1133646us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.051323 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:51.051366 (+    43us) raft_consensus.cc:1671] finished
0426 13:19:51.051378 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.051531 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38445,"prepare.queue_time_us":31305,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":212056,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":8942,"spinlock_wait_cycles":715136,"thread_start_us":1747593,"threads_started":2,"wal-append.queue_time_us":647644}
I20240426 13:19:51.242650 25097 rpcz_store.cc:276] 0426 13:19:49.887692 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.887984 (+   292us) service_pool.cc:225] Handling call
0426 13:19:49.888322 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.888592 (+   270us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:49.903393 (+ 14801us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.073744 (+1170351us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.073896 (+   152us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:51.074023 (+   127us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.074181 (+   158us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.074476 (+   295us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1268788,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":310,"raft.queue_time_us":42768,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":6928,"thread_start_us":1169747,"threads_started":1}
W20240426 13:19:51.242451 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.983s	user 0.000s	sys 0.037s
W20240426 13:19:51.251200 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.000s	sys 0.038s
I20240426 13:19:51.225687 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:51.254803 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 188353us
I20240426 13:19:51.255074 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1851) took 2074 ms. Trace:
I20240426 13:19:51.255640 24956 rpcz_store.cc:276] 0426 13:19:49.180169 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.181754 (+  1585us) service_pool.cc:225] Handling call
0426 13:19:51.254698 (+2072944us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.254872 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1405855488}
W20240426 13:19:51.242169 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 186106us
I20240426 13:19:51.241262 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1922) took 3959 ms. Trace:
I20240426 13:19:51.258086 25031 rpcz_store.cc:276] 0426 13:19:47.281368 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.792351 (+510983us) service_pool.cc:225] Handling call
0426 13:19:47.849365 (+ 57014us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.857063 (+  7698us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.857105 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.857608 (+   503us) log.cc:842] Serialized 26 byte log entry
0426 13:19:47.973300 (+115692us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:47.973342 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:47.973410 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:51.178884 (+3205474us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:51.240315 (+ 61431us) raft_consensus.cc:1671] finished
0426 13:19:51.240387 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.240935 (+   548us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7307,"raft.queue_time_us":157732,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":365789440,"thread_start_us":80053,"threads_started":1,"wal-append.queue_time_us":730549}
I20240426 13:19:51.259668 26824 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.259860 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 403) took 2638 ms. Trace:
W20240426 13:19:51.260278 26679 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.012s	sys 0.032s
I20240426 13:19:51.266337 24950 rpcz_store.cc:276] 0426 13:19:48.621704 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.630346 (+  8642us) service_pool.cc:225] Handling call
0426 13:19:48.653552 (+ 23206us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:48.742455 (+ 88903us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:48.742496 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:48.742545 (+    49us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:48.742577 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.259411 (+2516834us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.259484 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.259701 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165910272}
I20240426 13:19:51.285998 27360 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:51.286226 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.287230 25031 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 7
W20240426 13:19:51.288272 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 64296.5us
W20240426 13:19:51.288475 26679 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.014s	sys 0.032s
I20240426 13:19:51.294250 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 580) took 1406 ms. Trace:
I20240426 13:19:51.294842 25620 rpcz_store.cc:276] 0426 13:19:49.887433 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.887551 (+   118us) service_pool.cc:225] Handling call
0426 13:19:49.887831 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:51.132689 (+1244858us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.293560 (+160871us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.293953 (+   393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2843583744}
I20240426 13:19:51.295331 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.297220 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 550) took 2062 ms. Trace:
I20240426 13:19:51.297713 25623 rpcz_store.cc:276] 0426 13:19:49.234562 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.816690 (+582128us) service_pool.cc:225] Handling call
0426 13:19:49.817061 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.134122 (+317061us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.296653 (+1162531us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.296983 (+   330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541475072}
W20240426 13:19:51.297600 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52008.8us
I20240426 13:19:51.227926 24921 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:51.305055 27481 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:51.305267 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 77827.8us
I20240426 13:19:51.305742 27481 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.306386 27481 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:51.306442 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:51.299779 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.287134 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1425) took 2343 ms. Trace:
I20240426 13:19:51.312633 25099 rpcz_store.cc:276] 0426 13:19:48.944014 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.944163 (+   149us) service_pool.cc:225] Handling call
0426 13:19:48.944426 (+   263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.812742 (+868316us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.262728 (+1449986us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.286890 (+ 24162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1912283520}
W20240426 13:19:51.316529 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58378.8us
I20240426 13:19:51.325805 27492 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:51.331168 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.339193 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.343312 25098 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:51.338272 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.343840 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:51.356184 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.333825 27492 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.360095 27505 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.361178 27496 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20240426 13:19:51.361085 27505 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.362929 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1947) took 2145 ms. Trace:
I20240426 13:19:51.363512 25027 rpcz_store.cc:276] 0426 13:19:49.217221 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.709063 (+1491842us) service_pool.cc:225] Handling call
0426 13:19:50.709375 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.731935 (+ 22560us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.362223 (+630288us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.362691 (+   468us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50729856}
I20240426 13:19:51.368729 27475 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.368955 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.367916 25105 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.370138 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.372061 27492 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.375929 27498 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:51.377074 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46156.9us
I20240426 13:19:51.379936 25099 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.379206 27498 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240426 13:19:51.383913 27505 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.384156 27507 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:51.384802 27507 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.385207 27507 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.388340 26713 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:51.388823 26713 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.389245 26713 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:51.390218 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1902) took 5383 ms (client timeout 1504 ms). Trace:
W20240426 13:19:51.390717 25025 rpcz_store.cc:269] 0426 13:19:46.006668 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.006811 (+   143us) service_pool.cc:225] Handling call
0426 13:19:51.389761 (+5382950us) spinlock_profiling.cc:243] Waited 2.76 s on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.389978 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6341508864}
I20240426 13:19:51.377419 26969 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.393343 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1444) took 2417 ms. Trace:
I20240426 13:19:51.394035 25023 rpcz_store.cc:276] 0426 13:19:48.975678 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.525124 (+1549446us) service_pool.cc:225] Handling call
0426 13:19:50.526951 (+  1827us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.588279 (+ 61328us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.392756 (+804477us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.393138 (+   382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126969344}
I20240426 13:19:51.394724 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1437) took 2465 ms. Trace:
I20240426 13:19:51.395157 25029 rpcz_store.cc:276] 0426 13:19:48.929675 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.390892 (+1461217us) service_pool.cc:225] Handling call
0426 13:19:50.391303 (+   411us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.585667 (+194364us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.394310 (+808643us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.394542 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437067648}
I20240426 13:19:51.396327 25097 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:51.357331 27474 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.400527 27093 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.401561 27093 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.399638 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1920) took 4155 ms. Trace:
I20240426 13:19:51.401028 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.413097 25103 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:51.413838 27474 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.415823 27350 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:51.416258 27350 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:51.417382 27350 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.420563 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80304.4us
I20240426 13:19:51.421952 25102 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.422360 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:51.425251 27310 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:51.426769 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47496.1us
I20240426 13:19:51.427204 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1426) took 2482 ms. Trace:
I20240426 13:19:51.427623 25101 rpcz_store.cc:276] 0426 13:19:48.944353 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.944494 (+   141us) service_pool.cc:225] Handling call
0426 13:19:51.426762 (+2482268us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54003b06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.427012 (+   250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832645888}
I20240426 13:19:51.432457 25025 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:51.435084 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92332.5us
I20240426 13:19:51.436609 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 541) took 2995 ms. Trace:
I20240426 13:19:51.437057 25617 rpcz_store.cc:276] 0426 13:19:48.441063 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.460291 (+ 19228us) service_pool.cc:225] Handling call
0426 13:19:48.546765 (+ 86474us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:49.021202 (+474437us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.436005 (+2414803us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.436409 (+   404us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":902578048}
W20240426 13:19:51.437788 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 374) took 4296 ms (client timeout 1990 ms). Trace:
W20240426 13:19:51.438192 25619 rpcz_store.cc:269] 0426 13:19:47.140951 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.141078 (+   127us) service_pool.cc:225] Handling call
0426 13:19:51.437432 (+4296354us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54002a58d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.437598 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4260871936}
I20240426 13:19:51.443681 27496 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.444397 27496 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.369701 27475 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.446630 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1442) took 2480 ms (client timeout 2817 ms). Trace:
W20240426 13:19:51.447221 25026 rpcz_store.cc:269] 0426 13:19:48.966209 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.835219 (+869010us) service_pool.cc:225] Handling call
0426 13:19:51.446161 (+1610942us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54001e4954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.446384 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311917440}
I20240426 13:19:51.465354 25031 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:51.465780 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 166194us
I20240426 13:19:51.469674 27490 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.465735 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.469863 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.474372 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:51.476259 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54091.9us
I20240426 13:19:51.404561 25024 rpcz_store.cc:276] 0426 13:19:47.244312 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.630933 (+386621us) service_pool.cc:225] Handling call
0426 13:19:47.639948 (+  9015us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.550949 (+2911001us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b58004e05b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:19:50.607790 (+ 56841us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:51.228006 (+620216us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.365133 (+137127us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.370133 (+  5000us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:51.377932 (+  7799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.398974 (+ 21042us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.399577 (+   603us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8301,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":14415,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"spinlock_wait_cycles":374286080,"thread_start_us":733546,"threads_started":2}
I20240426 13:19:51.480513 27499 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.479811 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:51.481619 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.483412 27482 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:51.483628 27490 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:51.483738 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.487929 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:51.485199 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 331) took 2478 ms (client timeout 1525 ms). Trace:
I20240426 13:19:51.484390 26969 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:51.507776 25022 rpcz_store.cc:269] 0426 13:19:49.006577 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.014708 (+  8131us) service_pool.cc:225] Handling call
0426 13:19:51.484756 (+2470048us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.484947 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498607360}
I20240426 13:19:51.515491 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.516875 25098 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:51.520186 26679 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:51.520807 26679 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.521209 26679 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.506078 27481 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.505234 27093 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:51.525490 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57284.8us
I20240426 13:19:51.527004 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.527021 27492 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:51.528136 27482 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.529500 27481 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.532837 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49268.7us
I20240426 13:19:51.535897 27386 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.534303 27492 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:51.535437 27482 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.487269 27474 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.538296 27107 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.537765 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:51.540274 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:51.535359 25616 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.707s	user 0.011s	sys 0.007s
I20240426 13:19:51.544652 27271 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:51.545082 27492 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.546053 27470 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.556072 25096 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.556911 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:51.558933 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.559619 27379 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:51.560166 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.542263 27350 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:51.487591 27495 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.561857 25096 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.552619 27107 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.571952 25025 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:51.602838 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.600489 25098 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:51.606683 25098 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.562733 27379 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.610245 25619 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:51.611006 25028 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.003s
I20240426 13:19:51.611402 25028 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:51.612406 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6231911789999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
I20240426 13:19:51.616669 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:51.611713 27107 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.610270 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 547) took 2426 ms. Trace:
I20240426 13:19:51.626327 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.578014 27475 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.630806 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72090.3us
I20240426 13:19:51.628190 27416 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.632234 27416 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.633016 27496 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.633193 27416 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.634727 27416 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.635143 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.635918 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.636085 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:51.636400 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.636808 24953 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:51.637827 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.638533 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.636401 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.640522 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:51.640792 24950 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.641976 25096 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.616761 27327 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.643597 27327 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.645407 27426 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:51.645889 27426 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.646728 27426 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.618065 27379 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.647797 27327 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.648785 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60734.8us
I20240426 13:19:51.643105 24995 leader_election.cc:398] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:19:51.650501 24995 leader_election.cc:401] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:51.577327 25616 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:51.640302 25098 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.630818 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93364.4us
I20240426 13:19:51.596638 27150 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:51.661139 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 179903us
I20240426 13:19:51.661309 25102 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:51.562366 26713 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.675159 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1412) took 4479 ms (client timeout 1555 ms). Trace:
W20240426 13:19:51.675668 25102 rpcz_store.cc:269] 0426 13:19:47.187953 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.188109 (+   156us) service_pool.cc:225] Handling call
0426 13:19:49.253042 (+2064933us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b5400450720. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902adae1 00007fda9030e4b1 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2
0426 13:19:51.420873 (+2167831us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b580068d8b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:19:51.667541 (+246668us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1260481664}
I20240426 13:19:51.674204 27513 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.680002 24953 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:51.631966 27510 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:51.659014 27150 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:51.683243 25098 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.687242 27310 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.687496 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.667286 27386 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:51.619599 25615 rpcz_store.cc:276] 0426 13:19:49.183353 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.183534 (+   181us) service_pool.cc:225] Handling call
0426 13:19:49.219519 (+ 35985us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.219730 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:49.222905 (+  3175us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.375108 (+2152203us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.562859 (+187751us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.575151 (+ 12292us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:51.578600 (+  3449us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.599685 (+ 21085us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.610099 (+ 10414us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":464,"prepare.queue_time_us":133861,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":73,"raft.queue_time_us":72269,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":9682,"thread_start_us":2188912,"threads_started":4,"wal-append.queue_time_us":737147,"wal-append.run_cpu_time_us":1164,"wal-append.run_wall_time_us":1005237}
W20240426 13:19:51.695436 27461 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.016s	sys 0.019s
I20240426 13:19:51.694355 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.681650 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 5 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:51.698168 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:51.699489 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.689849 27150 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:51.711150 25069 connection.cc:232] RPC call timeout handler was delayed by -1.2973911789999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20240426 13:19:51.711949 25096 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.713012 27526 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:51.713171 27434 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.713678 27526 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.713714 27434 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.712193 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50235.3us
I20240426 13:19:51.689164 27472 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.717394 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.706571 24955 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.722365 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:51.706866 27461 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.017s	sys 0.020s
I20240426 13:19:51.712424 27498 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:51.732241 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62268.3us
I20240426 13:19:51.716009 27526 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.753595 26713 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.754287 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.755059 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:51.756605 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.757447 26679 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.758582 24854 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 272) took 18240 ms (client timeout 15000 ms). Trace:
I20240426 13:19:51.759483 26679 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.760421 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.761399 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:51.762358 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.763043 25102 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.763615 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.764528 27434 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.765127 25101 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:51.764535 25099 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.765830 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:51.766968 27386 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:51.767750 27386 consensus_queue.cc:1225] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:51.768326 27386 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:51.768965 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.767587 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:51.679023 27513 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.775137 27475 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.775595 27475 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.778407 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:51.768647 27386 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:51.800454 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88458.2us
I20240426 13:19:51.804756 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.777045 24953 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:51.809890 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 201555us
I20240426 13:19:51.814738 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.795241 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.734283 27508 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
W20240426 13:19:51.828055 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49599.9us
W20240426 13:19:51.826349 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58915.7us
W20240426 13:19:51.833667 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109484us
I20240426 13:19:51.811893 27472 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:51.832623 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.813534 27386 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.842926 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.849830 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:51.777660 27498 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.849984 27451 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20240426 13:19:51.855322 27350 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:51.855755 27472 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.862072 27416 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:51.874047 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135322us
W20240426 13:19:51.906129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94000us
W20240426 13:19:51.758962 24854 rpcz_store.cc:269] 0426 13:19:33.518193 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.527156 (+  8963us) service_pool.cc:225] Handling call
0426 13:19:51.758346 (+18231190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:46.050869 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:46.051519 (+   650us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:46.051574 (+    55us) write_op.cc:427] Acquired schema lock
0426 13:19:46.051594 (+    20us) tablet.cc:660] Decoding operations
0426 13:19:46.052107 (+   513us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:46.052186 (+    79us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:46.052215 (+    29us) tablet.cc:683] Acquiring locks for 2 operations
0426 13:19:46.052682 (+   467us) tablet.cc:699] Row locks acquired
0426 13:19:46.052716 (+    34us) write_op.cc:260] PREPARE: finished
0426 13:19:46.052879 (+   163us) write_op.cc:270] Start()
0426 13:19:46.053022 (+   143us) write_op.cc:275] Timestamp: P: 1714137586052815 usec, L: 0
0426 13:19:46.053052 (+    30us) op_driver.cc:331] REPLICATION: starting
0426 13:19:46.156550 (+103498us) log.cc:842] Serialized 917 byte log entry
0426 13:19:47.040220 (+883670us) op_driver.cc:444] REPLICATION: finished
0426 13:19:47.671830 (+631610us) write_op.cc:300] APPLY: starting
0426 13:19:47.723944 (+ 52114us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:47.775872 (+ 51928us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:47.778046 (+  2174us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:47.778479 (+   433us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:47.778671 (+   192us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0426 13:19:47.778739 (+    68us) write_op.cc:311] APPLY: finished
0426 13:19:47.779022 (+   283us) log.cc:842] Serialized 29 byte log entry
0426 13:19:47.779255 (+   233us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:47.779353 (+    98us) write_op.cc:446] Released schema lock
0426 13:19:47.779411 (+    58us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:47.779741 (+   330us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:47.779783 (+    42us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4200839,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":564169,"apply.run_cpu_time_us":11354,"apply.run_wall_time_us":130817,"num_ops":2,"prepare.queue_time_us":281434,"prepare.run_cpu_time_us":18948,"prepare.run_wall_time_us":233205,"raft.queue_time_us":353033,"raft.run_cpu_time_us":29712,"raft.run_wall_time_us":497397,"replication_time_us":966628,"thread_start_us":814797,"threads_started":4,"wal-append.queue_time_us":227854,"wal-append.run_cpu_time_us":27520,"wal-append.run_wall_time_us":2374020}]]}
I20240426 13:19:51.871750 27350 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.874910 27508 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:51.940917 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.944634 27410 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:51.945289 27410 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.946326 27410 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.867504 27475 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.948311 27487 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20240426 13:19:51.948877 27475 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.950078 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.951062 26996 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.951506 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.932150 27470 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.954330 27470 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.956465 27470 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:51.957087 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132791us
I20240426 13:19:51.957521 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.960325 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.960940 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.961061 27498 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.961586 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:51.962116 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.962904 25024 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:51.899343 26679 raft_consensus.cc:2741] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20240426 13:19:51.963104 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 140117us
I20240426 13:19:51.930912 24853 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.909742 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:51.961108 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.969601 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.932472 27350 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.974275 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.973300 25098 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:51.902781 27416 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.988909 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.989974 25615 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:51.985273 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.973613 27508 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.982074 24853 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:51.997949 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.982806 26969 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.982960 26824 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.980077 27474 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:51.987147 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 126216us
W20240426 13:19:51.994338 27507 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.609s	user 0.014s	sys 0.006s
I20240426 13:19:51.988312 27331 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.980799 27310 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.010998 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.011833 27507 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.013801 27507 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.014240 24947 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.015619 27327 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:52.016142 27486 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.971905 27492 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.017282 27327 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:52.018666 27327 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:52.016245 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:52.021368 27393 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:52.021742 27393 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.022058 27393 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:52.024134 27499 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:52.024703 27499 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:52.025658 27499 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.028594 27472 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.029526 27492 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.039280 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.039990 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:52.006919 26969 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.040951 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.041357 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.041738 26969 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:52.042229 24949 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:52.042945 24953 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:52.042033 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.044430 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.037837 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:52.045507 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:52.049785 27438 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.050078 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.050320 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.047024 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:52.050935 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.051384 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.051481 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.052311 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:52.052585 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.053638 24956 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:52.053659 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.054849 27474 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.049785 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.055960 24951 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:52.054407 27513 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.050221 27438 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.062657 27438 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:52.063084 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 93642.6us
I20240426 13:19:52.063892 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.063287 27508 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.066817 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.067432 25025 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:52.067811 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56915.4us
I20240426 13:19:52.068368 27513 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.051578 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:52.068802 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.071219 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.051815 25618 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:52.057610 25028 raft_consensus.cc:2362] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5. Re-sending same reply.
I20240426 13:19:52.075181 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.075781 25028 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.076398 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.080675 27474 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.082926 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:19:52.058019 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106644us
I20240426 13:19:51.982687 27271 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20240426 13:19:52.060048 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51643.1us
I20240426 13:19:52.085238 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.056818 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:52.085323 25028 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:51.997459 27496 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20240426 13:19:52.082538 27438 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.086570 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.087746 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.088286 27438 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.088764 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.089380 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.090876 25619 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:52.019182 27327 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.998404 27416 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.097501 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:52.103960 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69591.8us
I20240426 13:19:52.097066 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:52.112946 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44289.1us
I20240426 13:19:52.113893 25030 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:52.116520 24949 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:52.043499 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.129238 27331 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:52.129796 27496 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.130264 27496 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:52.130681 27496 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.059772 26824 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.133280 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.136735 27426 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:52.140161 24953 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:52.117861 27507 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20240426 13:19:52.142128 27507 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:52.141856 27350 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.142549 27507 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:52.144266 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.129829 27331 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.145038 27331 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.145227 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.145505 27350 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.147439 24948 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:52.148630 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.149878 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.150624 24948 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:52.150777 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:52.151445 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.151767 27376 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.152216 27376 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.152608 27376 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.093271 27107 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.034999 27495 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20240426 13:19:52.123729 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.152021 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:52.166584 25028 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:52.155737 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.169575 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.175163 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.175985 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:52.143429 27507 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.180277 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:52.184218 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122588us
I20240426 13:19:52.185169 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:52.186965 25030 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 13:19:52.187482 25030 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 2
I20240426 13:19:52.133909 26824 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.164655 27475 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.168704 27107 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:52.192047 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106931us
I20240426 13:19:52.198731 25030 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 13:19:52.208158 27474 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:52.208726 27474 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.209120 27474 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:52.199213 27327 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.199179 25030 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:52.183838 27434 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.214287 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:52.219504 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.187923 27490 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:52.221470 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77290.8us
I20240426 13:19:52.216995 26969 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.236224 27434 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.201608 27107 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.218294 25028 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:52.254478 25028 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:19:52.254885 25028 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:52.257890 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:52.261415 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60940us
I20240426 13:19:52.262818 24947 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.253947 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:52.289363 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88470.1us
I20240426 13:19:52.241171 26969 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.253100 27434 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:52.267323 27150 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.000s
W20240426 13:19:52.300833 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83019.7us
I20240426 13:19:52.202410 27393 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.327462 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:52.302489 26969 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.335253 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:52.293210 27482 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:52.319355 27487 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:52.354722 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102512us
I20240426 13:19:52.305959 27093 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20240426 13:19:52.359617 27093 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:52.360059 27093 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:52.360394 27591 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20240426 13:19:52.361338 27501 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.361217 27093 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.362006 27303 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.362461 27303 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.362901 27303 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:52.358613 27600 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20240426 13:19:52.365326 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.367164 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.368182 27426 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:52.368816 27426 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:52.369274 27426 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:52.370124 27426 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.268507 27490 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.305399 27507 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.340384 27510 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:52.385310 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:52.386205 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.339933 27327 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.399807 27486 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:52.401091 27486 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:52.401543 27486 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:52.402505 27486 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.311182 27150 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.405827 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.406754 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.411715 25030 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:52.355635 27451 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.416076 25102 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:52.419421 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:52.428555 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.429854 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.430987 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.431901 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.371753 27376 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.448045 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.448876 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.455282 25100 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.444867 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.432497 27451 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.422526 27498 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:52.391804 27492 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20240426 13:19:52.434517 26679 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.375499 27386 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.416734 27513 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 13:19:52.475785 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.476675 25097 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:52.455575 27093 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.465250 27451 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.467378 26679 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.464699 27498 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:52.509675 25030 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:52.494767 27379 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.822s	user 0.013s	sys 0.000s
I20240426 13:19:52.512495 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:52.520740 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.035s	sys 0.041s
W20240426 13:19:52.521171 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.035s	sys 0.041s
I20240426 13:19:52.523311 25101 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.521009 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.526484 25098 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:52.525336 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.538329 25097 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.508267 26679 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.508473 27498 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.542407 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.549080 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:52.572077 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144581us
I20240426 13:19:52.532368 27490 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.530330 27393 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.582429 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.583148 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.584237 24947 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:52.584380 27331 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.585460 24947 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:52.585780 26969 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.589424 27487 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:52.590549 25030 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:52.601541 27376 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:52.605170 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.607995 27416 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.586349 27331 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.536120 27461 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:52.615399 25101 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:52.557533 27475 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:52.633026 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:52.638337 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.524883 27379 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.628834 27416 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:52.643314 27508 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.013s	sys 0.021s
I20240426 13:19:52.641322 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.649137 25096 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.650228 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:52.646914 24952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.027s	sys 0.038s
W20240426 13:19:52.686959 24952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.027s	sys 0.038s
W20240426 13:19:52.644740 27499 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.016s	sys 0.014s
I20240426 13:19:52.643136 27376 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:52.661954 27508 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.013s	sys 0.021s
W20240426 13:19:52.645164 27410 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.014s	sys 0.017s
I20240426 13:19:52.648185 27475 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.648559 27150 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:52.691052 27499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.017s	sys 0.014s
W20240426 13:19:52.702834 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.010s	sys 0.027s
I20240426 13:19:52.686724 26969 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.692426 27376 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:52.737673 24853 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.011s	sys 0.032s
W20240426 13:19:52.740121 24853 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.012s	sys 0.032s
W20240426 13:19:52.730875 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.010s	sys 0.027s
W20240426 13:19:52.700711 27410 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.015s	sys 0.017s
I20240426 13:19:52.656440 27474 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:52.746721 27605 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20240426 13:19:52.647373 27461 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.732738 27386 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.742138 27376 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:52.748214 27606 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s	user 0.001s	sys 0.001s
I20240426 13:19:52.738270 27150 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20240426 13:19:52.812800 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 476265us
W20240426 13:19:52.829582 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 451108us
I20240426 13:19:52.868192 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1615) took 1094 ms. Trace:
I20240426 13:19:52.824646 27482 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:52.868821 25100 rpcz_store.cc:276] 0426 13:19:51.759675 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.760119 (+   444us) service_pool.cc:225] Handling call
0426 13:19:52.853761 (+1093642us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:52.853935 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1575075200}
I20240426 13:19:52.878497 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.879191 24947 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:52.880494 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1871) took 1182 ms. Trace:
I20240426 13:19:52.811177 27410 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20240426 13:19:52.880901 24950 rpcz_store.cc:276] 0426 13:19:51.690616 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.690738 (+   122us) service_pool.cc:225] Handling call
0426 13:19:51.699262 (+  8524us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:51.709668 (+ 10406us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:51.709709 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:51.710290 (+   581us) log.cc:842] Serialized 26 byte log entry
0426 13:19:51.994590 (+284300us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:51.994622 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.994682 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.873018 (+878336us) raft_consensus.cc:1671] finished
0426 13:19:52.873043 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.873220 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2667,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":283931,"threads_started":1,"wal-append.queue_time_us":744658}
I20240426 13:19:52.838577 27510 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:52.886476 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.886457 27507 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.815824 27303 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.855559 27499 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.820792 26996 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
W20240426 13:19:52.882951 27418 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.020s	sys 0.023s
I20240426 13:19:52.903542 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.878944 27379 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:52.922309 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 400530us
I20240426 13:19:52.906064 27410 consensus_queue.cc:1225] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:19:52.919725 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.928247 24950 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.933483 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:52.922678 27418 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.020s	sys 0.024s
I20240426 13:19:52.924690 27416 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.946413 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.926517 27438 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:52.933821 27350 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.953457 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.955252 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.957304 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.958812 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.959621 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1953) took 1002 ms. Trace:
I20240426 13:19:52.959982 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.965281 27451 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.966928 25099 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.967012 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1956) took 1008 ms. Trace:
I20240426 13:19:52.967839 27451 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.960002 25029 rpcz_store.cc:276] 0426 13:19:51.957446 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.960174 (+  2728us) service_pool.cc:225] Handling call
0426 13:19:51.960471 (+   297us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:51.987839 (+ 27368us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:51.987879 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:51.988380 (+   501us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.521671 (+533291us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:52.912218 (+390547us) log.cc:842] Serialized 14 byte log entry
0426 13:19:52.923773 (+ 11555us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.924268 (+   495us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.959392 (+ 35124us) raft_consensus.cc:1671] finished
0426 13:19:52.959422 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.959566 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24969,"prepare.queue_time_us":356016,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":34361,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":11688,"thread_start_us":891332,"threads_started":2,"wal-append.queue_time_us":778420}
I20240426 13:19:52.978281 25029 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.967552 25027 rpcz_store.cc:276] 0426 13:19:51.958442 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.958570 (+   128us) service_pool.cc:225] Handling call
0426 13:19:51.958918 (+   348us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.005190 (+ 46272us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.005230 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.060983 (+ 55753us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.220737 (+159754us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b180043bca0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:52.573673 (+352936us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.573724 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.573803 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.966665 (+392862us) raft_consensus.cc:1671] finished
0426 13:19:52.966693 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.966944 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37302,"raft.queue_time_us":44853,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":51718,"spinlock_wait_cycles":68600704,"thread_start_us":352757,"threads_started":1,"wal-append.queue_time_us":792752}
I20240426 13:19:52.996204 25027 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.936434 27501 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:53.031450 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 393364us
W20240426 13:19:53.032374 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 621852us
I20240426 13:19:52.923506 27499 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.037150 24921 connection.cc:232] RPC call timeout handler was delayed by -1.7364382590000105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20240426 13:19:53.037647 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5960382590000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17110000000000003s for a response
I20240426 13:19:52.930217 27507 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.037935 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4142382589999443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20240426 13:19:53.038870 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3892003540000815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
W20240426 13:19:53.039335 24921 connection.cc:232] RPC call timeout handler was delayed by -1.2621382590000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20240426 13:19:53.039530 27507 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.938372 27379 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20240426 13:19:52.955408 27410 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:52.952576 27474 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.953058 27416 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.035241 27501 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.039659 27303 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.047019 27638 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.043709 27107 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.743s	user 0.007s	sys 0.002s
I20240426 13:19:53.026916 27350 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.060125 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1955) took 1087 ms. Trace:
I20240426 13:19:53.042802 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.075387 25031 rpcz_store.cc:276] 0426 13:19:51.958114 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.961176 (+  3062us) service_pool.cc:225] Handling call
0426 13:19:51.961457 (+   281us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.008184 (+ 46727us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.008216 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.046699 (+ 38483us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.749650 (+702951us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.749702 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.749797 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.044897 (+295100us) raft_consensus.cc:1671] finished
0426 13:19:53.044925 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.045070 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53574,"raft.queue_time_us":44744,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":19837,"thread_start_us":571571,"threads_started":1,"wal-append.queue_time_us":807718}
I20240426 13:19:53.072451 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.076294 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.082507 25027 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.063963 24950 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:19:53.092584 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57120.1us
I20240426 13:19:52.951458 27510 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.095504 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.096019 27350 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.096146 25025 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:53.073863 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.097570 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.946543 27482 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.098662 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.099414 25026 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.072674 26679 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.046365 27461 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:53.102491 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60155.9us
I20240426 13:19:53.102746 27526 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:53.103281 27526 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.103652 27647 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.104542 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.105059 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.105845 25030 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.106657 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.107654 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.068758 27416 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.109238 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.110458 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:53.057847 27434 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.731s	user 0.000s	sys 0.008s
I20240426 13:19:53.112282 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.112421 27434 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.113061 25029 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20240426 13:19:53.113663 27496 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.983s	user 0.010s	sys 0.001s
I20240426 13:19:53.113829 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.064405 27648 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:53.114048 27496 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.115059 27622 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.369s	user 0.002s	sys 0.000s
I20240426 13:19:53.115275 24955 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.117058 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.117542 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1960) took 1045 ms. Trace:
I20240426 13:19:53.118360 25027 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.118628 27496 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.112757 27393 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.131491 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.120057 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.123188 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:53.124163 25029 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.123648 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:53.121168 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.138435 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.144904 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1873) took 1051 ms. Trace:
I20240426 13:19:53.145308 24955 rpcz_store.cc:276] 0426 13:19:52.093874 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.094039 (+   165us) service_pool.cc:225] Handling call
0426 13:19:52.124333 (+ 30294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:53.115645 (+991312us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:53.144692 (+ 29047us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b54000794d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:53.144851 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2278601600}
I20240426 13:19:53.117920 25024 rpcz_store.cc:276] 0426 13:19:52.072494 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.079277 (+  6783us) service_pool.cc:225] Handling call
0426 13:19:52.079580 (+   303us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.157402 (+ 77822us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.157446 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.173079 (+ 15633us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.464248 (+291169us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:52.464303 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.464392 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.114438 (+650046us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.117237 (+  2799us) raft_consensus.cc:1671] finished
0426 13:19:53.117258 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.117480 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59656,"raft.queue_time_us":83583,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7519,"spinlock_wait_cycles":7889792,"thread_start_us":253234,"threads_started":1,"wal-append.queue_time_us":551388}
I20240426 13:19:53.133096 24947 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:53.149674 24949 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:53.149807 24948 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.149870 25031 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.157016 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.058174 27492 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.157892 24952 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.131525 27393 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:53.162238 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87530.3us
I20240426 13:19:53.163182 25024 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:53.119462 27107 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.168118 27648 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.184103 25102 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 13:19:53.146428 27618 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.400s	user 0.003s	sys 0.001s
I20240426 13:19:53.047518 27498 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.192090 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:53.168601 27472 raft_consensus.cc:659] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.169617 24956 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:53.056365 27499 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.066403 27410 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.174909 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:53.181648 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.200495 24995 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:19:53.200938 24995 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20240426 13:19:53.185436 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 239127us
W20240426 13:19:53.147258 27327 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.738s	user 0.008s	sys 0.002s
I20240426 13:19:53.168457 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:53.197751 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71000.5us
I20240426 13:19:53.198120 25096 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:53.247187 25022 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:53.247542 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 115282us
I20240426 13:19:53.180975 27492 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.249317 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:53.203341 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.205443 25026 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20240426 13:19:53.253623 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53296.8us
I20240426 13:19:53.192481 26824 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.260445 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:19:53.262562 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59261.6us
I20240426 13:19:53.263309 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.263978 27632 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.265514 27631 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.266005 27631 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.267297 25029 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:53.149947 27526 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.281800 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.272470 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.272909 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.300424 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.300802 27379 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.290529 27513 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:53.301716 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.301961 27513 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.302309 25105 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.302932 27513 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.304653 25104 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:53.272542 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.305644 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.306149 25104 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:53.307297 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.308445 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.157447 27376 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.295588 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1961) took 1222 ms (client timeout 1607 ms). Trace:
W20240426 13:19:53.309844 25022 rpcz_store.cc:269] 0426 13:19:52.072879 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.076163 (+  3284us) service_pool.cc:225] Handling call
0426 13:19:53.295326 (+1219163us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001e4e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:53.295505 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586791296}
I20240426 13:19:53.310632 27376 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.309856 25102 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:53.310731 27638 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:53.267962 27631 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.312213 27379 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.312605 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.312700 27379 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.313086 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.313802 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.314162 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.313923 27379 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.315166 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.315276 27107 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.316355 27271 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:53.204764 27501 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.317972 27107 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.318529 27107 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.318827 27501 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.315954 25102 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:53.273293 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.286734 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:53.290208 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.301178 25097 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.322124 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:53.323561 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:53.322373 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 135863us
I20240426 13:19:53.274819 25099 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:19:53.325541 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.326488 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:53.201409 27416 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.308844 27495 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.355103 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.355734 25100 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.347992 25102 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:53.347306 25101 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:53.346976 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98991.4us
I20240426 13:19:53.339100 24952 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:53.333292 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.354581 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.363080 24956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.363034 24948 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.270759 27434 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:53.359863 27150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.071s	sys 0.043s
I20240426 13:19:53.381431 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.381489 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:53.328008 27648 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.286659 27637 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:19:53.347630 27418 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:53.374965 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.371048 27416 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:53.385551 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1963) took 1312 ms. Trace:
I20240426 13:19:53.394320 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:53.359670 27379 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.362566 27526 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:53.269197 27498 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.400506 25099 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:53.297201 26679 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.347151 27638 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20240426 13:19:53.409323 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.354501 27631 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240426 13:19:53.402701 25023 rpcz_store.cc:276] 0426 13:19:52.073502 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.077732 (+  4230us) service_pool.cc:225] Handling call
0426 13:19:52.078027 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.078304 (+   277us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.078327 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.078708 (+   381us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.536613 (+457905us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.536693 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.536825 (+   132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.111476 (+574651us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.374464 (+262988us) raft_consensus.cc:1671] finished
0426 13:19:53.374488 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.385478 (+ 10990us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13713280,"thread_start_us":457536,"threads_started":1,"wal-append.queue_time_us":667444}
I20240426 13:19:53.286361 27393 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.327288 26824 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.438486 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.427810 27649 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.408867 27628 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:53.381002 27487 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 13:19:53.409624 27416 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:53.292475 27410 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.442528 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.385318 27438 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20240426 13:19:53.450764 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89292.1us
I20240426 13:19:53.416589 27647 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.394958 27426 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:19:53.387066 27495 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.399741 27472 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
W20240426 13:19:53.452960 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93564.3us
I20240426 13:19:53.428408 27418 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:53.488063 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46920.6us
I20240426 13:19:53.488905 27418 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.489663 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.480834 27655 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:53.481634 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.491262 27495 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.452158 27416 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.460314 27376 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:53.471084 27651 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20240426 13:19:53.485036 27643 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:53.486258 27657 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:19:53.504253 27643 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:53.504724 27643 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.483747 25097 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:53.496382 24951 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.506160 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:53.508143 27249 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:53.508694 27249 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.509544 27249 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.460234 27510 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.498836 25100 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:53.502141 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.442523 27632 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:53.503017 27633 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.503372 27640 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.588201 27640 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.588652 27631 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.589294 27631 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:53.589004 27640 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.589740 27631 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.590209 27640 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.590658 27631 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.592171 27474 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:53.592530 27474 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:53.592870 27659 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
I20240426 13:19:53.593317 27474 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.594344 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164831us
I20240426 13:19:53.474910 27482 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.469144 27393 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:53.559377 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53347.5us
I20240426 13:19:53.559829 24956 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.417250 27492 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.468806 27649 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.515880 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:53.634034 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 131945us
W20240426 13:19:53.637904 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 148330us
I20240426 13:19:53.522416 27510 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.580706 27638 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.577631 27495 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.071s
I20240426 13:19:53.622056 27647 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.671198 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.638698 27474 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.663511 25105 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:53.594964 27633 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.640723 27107 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.653860 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:53.665043 27492 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.715468 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.684267 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.631347 27482 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.686434 27310 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.660208 27648 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 7 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:53.689365 27327 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.698601 27510 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.713461 25104 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:53.715601 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 96444.9us
I20240426 13:19:53.674638 27496 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:53.725061 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.732771 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.750630 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.715271 27501 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.732148 27496 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.774283 27496 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.730010 27327 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.749828 26824 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:53.778049 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93850.2us
I20240426 13:19:53.729533 27107 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.735661 27310 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.723183 27434 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 7 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:53.786585 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 270772us
I20240426 13:19:53.787813 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.725677 27643 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.781281 27376 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.797457 27513 raft_consensus.cc:2741] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240426 13:19:53.791936 27107 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.775934 27501 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.810956 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.838956 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:53.804553 27647 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.814265 25096 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:53.843986 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52591.5us
I20240426 13:19:53.800695 27628 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.797971 27496 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20240426 13:19:53.847100 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69570.4us
I20240426 13:19:53.810810 27676 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:53.828296 27379 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.840238 27495 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:53.855533 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105009us
I20240426 13:19:53.857970 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.849936 27310 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:53.861140 27666 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.861279 27310 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.861776 27310 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.861299 27628 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.861735 27666 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.863518 27376 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.837683 27633 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.848990 27526 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.916093 27526 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:53.916455 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68016.1us
I20240426 13:19:53.918049 27526 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:53.919135 27645 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.448s	user 0.005s	sys 0.002s
I20240426 13:19:53.919708 27482 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:53.930972 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 576999us
I20240426 13:19:53.931530 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1888) took 1317 ms. Trace:
I20240426 13:19:53.901058 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.920217 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.920374 26679 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:53.921687 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.922135 27643 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.940452 25104 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:19:53.922391 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.923143 27327 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.923059 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.944527 25025 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.925442 27416 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.926545 25027 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:53.901623 27379 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.950863 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.952067 27379 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.953032 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.931996 24954 rpcz_store.cc:276] 0426 13:19:52.614071 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.614859 (+   788us) service_pool.cc:225] Handling call
0426 13:19:52.641992 (+ 27133us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.644189 (+  2197us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.644227 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.654316 (+ 10089us) log.cc:842] Serialized 26 byte log entry
0426 13:19:53.131338 (+477022us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:53.131387 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:53.131457 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.918338 (+786881us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.931199 (+ 12861us) raft_consensus.cc:1671] finished
0426 13:19:53.931227 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.931467 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11602688,"thread_start_us":415535,"threads_started":1,"wal-append.queue_time_us":733308}
I20240426 13:19:53.956449 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:53.957145 27501 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.957381 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:53.892992 27651 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20240426 13:19:53.927230 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.901295 27676 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:53.927891 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:53.928436 27675 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:19:53.929064 27451 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:53.893949 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:53.931706 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:53.850164 27513 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.942169 25030 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.945763 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.943921 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.947453 25104 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:53.940155 27482 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.899361 27310 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.948194 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:53.912535 27669 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20240426 13:19:53.863816 27666 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.951736 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.970572 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.988546 25022 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:53.988829 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94944.3us
I20240426 13:19:53.989177 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.989950 25027 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.990352 27501 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:53.990876 27501 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:53.991921 25031 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.992550 25097 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:53.991626 27501 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.998404 27513 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.999614 25105 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:54.001363 27513 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.986562 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.870064 27674 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
I20240426 13:19:54.005743 27677 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:54.007536 27651 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.008229 27651 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.009096 25022 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:54.010756 27651 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.853971 27684 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.011799 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.012392 25023 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:54.012478 24951 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:53.985505 27451 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:53.962679 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 111342us
I20240426 13:19:54.014778 27451 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.022279 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.025945 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:19:54.026837 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.027571 25100 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.027727 27674 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.029995 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.033366 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.030696 27674 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.042276 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:54.043409 27651 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:54.043848 27651 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.044356 27651 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.048471 27327 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:54.051717 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.052425 24956 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:54.053303 27327 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:54.055509 26996 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:54.056005 26996 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:54.055596 27674 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.057112 26996 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.057451 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:54.073537 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 141908us
I20240426 13:19:53.949153 27633 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.075361 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.076289 24951 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.078404 25030 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.078795 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.079427 25097 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:19:54.076838 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90362.5us
I20240426 13:19:54.080579 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.081261 24948 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.081466 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.074317 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:54.084018 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.084590 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.084708 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:54.085278 24949 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.090390 24956 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:53.970342 27107 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:53.968657 27676 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.059666 27474 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.987463 27438 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.091167 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.139411 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.086357 27495 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.144100 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.088312 27492 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.078034 27647 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.084017 27327 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.152730 24954 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:19:54.142560 24949 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.152972 24948 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:54.154244 25029 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.102135 27684 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.156989 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.163574 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:54.166553 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:54.177188 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 72958.4us
W20240426 13:19:54.179297 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105051us
I20240426 13:19:54.189218 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.189293 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:54.181387 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.088467 27666 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.149801 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:54.199229 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 117801us
I20240426 13:19:54.187364 24956 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:54.144320 27676 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.190434 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.206966 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.190918 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.151877 27310 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.209043 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.209148 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.204444 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.212218 24955 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.147164 27107 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.173192 27093 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.194227 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.208910 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.224061 25101 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.205984 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.224543 24947 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:54.174582 27438 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.180160 27327 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.217836 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.230882 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.231246 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:54.231794 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.232583 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:54.235009 24949 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.240690 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.241549 25101 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:54.245599 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.246439 25029 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:54.252321 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.253043 25623 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:54.205166 27513 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.254324 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.256997 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.183804 27495 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:54.289922 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58759.1us
I20240426 13:19:54.290377 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.291767 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.292590 25030 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.258052 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:54.295948 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:54.297657 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90748.1us
I20240426 13:19:54.298139 24956 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:54.298529 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.300948 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.242414 27327 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.187322 27647 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.253090 27418 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.302811 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.240414 27682 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.219760 27651 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.309710 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.276631 27666 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.322197 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.304401 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.323315 25101 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:54.234517 27093 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.295382 27513 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.329331 24948 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:54.330603 24947 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:54.295814 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.306866 27643 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.307087 27303 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.306177 27379 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.320006 27434 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.335304 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.306949 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.336580 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.307725 25616 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:54.333933 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.345229 24956 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.319733 27416 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.224658 27684 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:54.344851 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 155749us
I20240426 13:19:54.348320 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.364892 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:54.355496 25097 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.357383 25619 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:54.360920 25030 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.362211 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.362622 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.367488 24950 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.337502 27492 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.302204 27687 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20240426 13:19:54.369779 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.377558 24947 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.374405 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.375587 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.383489 24955 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:54.385490 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.387308 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.388942 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.391057 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.400584 25101 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.408459 24953 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:54.412611 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 105742us
I20240426 13:19:54.416307 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.417315 27093 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:54.417752 25621 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:54.417827 27482 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.393627 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.438441 24950 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:54.439225 24950 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:54.447273 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.379596 27656 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:54.419821 25101 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20240426 13:19:54.420749 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.455456 25097 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:54.422569 24956 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.416852 27666 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.412006 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.402767 27631 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.375816 27416 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.375674 27501 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:54.397404 27677 raft_consensus.cc:2741] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20240426 13:19:54.418116 27680 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20240426 13:19:54.413101 27682 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.452086 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.470808 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.502209 25029 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.472201 27496 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.424331 27685 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:54.501272 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.342025 27107 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.449331 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.462839 27482 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.451431 27418 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.477330 24950 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.518435 24950 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:54.508390 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 113609us
I20240426 13:19:54.506626 25622 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:54.346459 27438 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.545537 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:54.538466 27271 raft_consensus.cc:1054] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Unable to promote non-voter 1590f80bdade41c6a755b8501c55fe16: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:54.501832 27682 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.538628 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.501000 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.505931 27303 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.540849 27416 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.517197 27685 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.557404 24950 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:54.569697 24950 pending_rounds.cc:76] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:54.570060 24950 pending_rounds.cc:97] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:54.570412 24950 raft_consensus.cc:2866] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:54.559093 25029 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20240426 13:19:54.561736 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109787us
I20240426 13:19:54.572528 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.574111 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.517442 27677 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.543812 27472 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.468463 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:54.545881 27486 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:54.542709 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
W20240426 13:19:54.584465 27692 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20240426 13:19:54.585003 27647 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.501305 27631 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.588013 27651 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.588114 27631 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.589323 27684 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:54.590183 27689 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20240426 13:19:54.590417 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.591795 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.592605 25101 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.562533 27682 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.599390 27303 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.543284 27513 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.558086 27638 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.609004 25024 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:54.550140 27379 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.580289 27310 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.612788 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:54.613612 27701 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20240426 13:19:54.569402 27495 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.616758 25622 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:54.542572 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.638657 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:54.615948 27496 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:54.674755 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 204023us
I20240426 13:19:54.663820 27434 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:54.661480 27645 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.080s	user 0.001s	sys 0.003s
I20240426 13:19:54.604271 27327 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.684933 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.694511 25102 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.665436 27495 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.666146 27643 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.603849 27677 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:54.701115 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 241718us
I20240426 13:19:54.664932 27492 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.717695 25030 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:54.721545 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.729176 24950 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:54.728886 25030 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.703075 27379 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.682421 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:54.733378 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.750285 24954 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.735188 25102 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:54.732928 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 321029us
I20240426 13:19:54.734397 25622 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:54.738163 27410 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.737735 27643 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.661617 27418 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.740082 27696 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:54.741536 25623 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:54.731444 25030 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.779742 25030 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.699177 27495 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.766317 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:54.782122 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.783439 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.790081 25022 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:19:54.789912 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:54.791478 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 217449us
I20240426 13:19:54.740748 27697 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.703194 27434 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.778044 27696 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:54.800071 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.784456 27495 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.803427 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:54.826817 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92474.3us
I20240426 13:19:54.792615 27416 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.790009 27107 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:54.857389 27656 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:54.781651 27633 consensus_queue.cc:1159] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 6 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:54.740511 27709 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.175s
I20240426 13:19:54.832429 27684 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.858857 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.870604 27656 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:54.837028 27482 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.874718 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:54.876657 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 76800.5us
I20240426 13:19:54.881237 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.882416 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.883534 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.885710 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 598) took 1272 ms. Trace:
I20240426 13:19:54.878463 27696 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:54.864872 27107 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.859016 27303 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.894490 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.895428 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.886150 25615 rpcz_store.cc:276] 0426 13:19:53.612923 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.613184 (+   261us) service_pool.cc:225] Handling call
0426 13:19:54.885039 (+1271855us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:54.885218 (+   179us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:54.885391 (+   173us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.885415 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:54.885458 (+    43us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:54.885502 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.885557 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:54.885658 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6044416}
I20240426 13:19:54.784353 27715 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20240426 13:19:54.891119 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.895740 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.896057 24950 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.875218 27416 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.900502 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.863834 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20240426 13:19:54.863740 27631 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:54.901825 27730 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.902942 27696 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:54.906261 27482 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.893083 27501 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:54.908859 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:19:54.943516 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:54.901089 27704 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
W20240426 13:19:54.945832 27474 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.787s	user 0.010s	sys 0.003s
I20240426 13:19:54.881294 27685 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:54.896443 27107 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:54.895917 27697 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.896895 27684 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.995852 27310 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.869308 27492 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.034611 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 159984us
I20240426 13:19:55.038236 24950 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:55.013206 27327 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.016482 27496 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:55.052119 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:55.057199 25101 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.946708 27416 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.028016 27474 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.938704 27303 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.048444 27685 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:55.049374 27697 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:55.050755 27651 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:19:55.141578 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71540.1us
I20240426 13:19:55.203116 27416 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.202244 27733 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20240426 13:19:55.039531 27684 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.149003 27631 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
W20240426 13:19:55.209909 27702 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.521s	user 0.000s	sys 0.000s
W20240426 13:19:55.212550 27662 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.744s	user 0.007s	sys 0.004s
I20240426 13:19:55.203397 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 897) took 1365 ms. Trace:
W20240426 13:19:55.140801 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197381us
I20240426 13:19:55.210565 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 361) took 1186 ms. Trace:
I20240426 13:19:55.066154 27677 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.148535 27704 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:19:55.033715 27687 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20240426 13:19:55.202968 27495 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.262550 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:55.263662 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1991) took 1326 ms. Trace:
I20240426 13:19:55.264226 27495 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.196645 27486 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.003s
I20240426 13:19:55.199878 27709 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.211230 27688 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.847s	user 0.000s	sys 0.001s
I20240426 13:19:55.265403 25103 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:55.265002 27709 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.040241 27715 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.266023 27715 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.267048 27709 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.264150 25026 rpcz_store.cc:276] 0426 13:19:53.937318 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.000506 (+ 63188us) service_pool.cc:225] Handling call
0426 13:19:54.000901 (+   395us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.001312 (+   411us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:54.224385 (+223073us) log.cc:842] Serialized 14 byte log entry
0426 13:19:54.618206 (+393821us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.707816 (+ 89610us) log.cc:842] Serialized 26 byte log entry
0426 13:19:54.707988 (+   172us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:54.708050 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.708135 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.263377 (+555242us) raft_consensus.cc:1671] finished
0426 13:19:55.263408 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.263593 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7605,"prepare.queue_time_us":251418,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":6640,"raft.queue_time_us":57448,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":10603,"thread_start_us":537856,"threads_started":3,"wal-append.queue_time_us":305534}
I20240426 13:19:55.267983 27715 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.207325 27682 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.000s
I20240426 13:19:55.056249 27699 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20240426 13:19:55.019945 27482 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.268708 27682 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.270248 27682 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.270453 27482 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.067844 27727 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20240426 13:19:55.256323 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.268424 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:55.262256 25027 rpcz_store.cc:276] 0426 13:19:54.024148 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.588264 (+564116us) service_pool.cc:225] Handling call
0426 13:19:54.588615 (+   351us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:54.588951 (+   336us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.588978 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:54.589511 (+   533us) log.cc:842] Serialized 52 byte log entry
0426 13:19:54.797702 (+208191us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:54.851276 (+ 53574us) log.cc:842] Serialized 14 byte log entry
0426 13:19:54.868150 (+ 16874us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.868249 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.210364 (+342115us) raft_consensus.cc:1671] finished
0426 13:19:55.210384 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.210513 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7177,"prepare.queue_time_us":54417,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":45171,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"spinlock_wait_cycles":189312,"thread_start_us":217013,"threads_started":2,"wal-append.queue_time_us":618266}
I20240426 13:19:55.071682 27740 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.137s
I20240426 13:19:55.273394 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.274201 25026 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:55.274089 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.261715 25028 rpcz_store.cc:276] 0426 13:19:53.838313 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.842086 (+  3773us) service_pool.cc:225] Handling call
0426 13:19:55.202798 (+1360712us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.203308 (+   510us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":367993216}
W20240426 13:19:55.279728 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 270653us
I20240426 13:19:55.280015 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:55.255793 27651 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:55.287626 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 257494us
I20240426 13:19:55.275540 27498 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.260464 27697 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.292668 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.281138 27699 consensus_queue.cc:1225] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240426 13:19:55.286566 27685 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.285996 27495 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:55.299432 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.323010 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.325366 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.313349 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.287209 27651 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:55.315240 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:55.294314 27687 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.336478 24954 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.000s
I20240426 13:19:55.336746 24947 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:55.336815 24954 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:55.294307 27647 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.337776 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.294435 27730 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.335134 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.304323 27715 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.322815 27710 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.344627 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46741.7us
I20240426 13:19:55.347352 24953 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:55.347652 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 404200us
I20240426 13:19:55.344123 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:55.325141 27510 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:55.351615 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2002) took 1015 ms. Trace:
I20240426 13:19:55.353935 25028 rpcz_store.cc:276] 0426 13:19:54.335807 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.313732 (+977925us) service_pool.cc:225] Handling call
0426 13:19:55.335999 (+ 22267us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.351390 (+ 15391us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.351565 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:55.291566 27746 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.200s
I20240426 13:19:55.336300 27482 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:55.329442 27740 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:55.357817 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:55.364384 25619 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:55.342808 27682 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.294955 27498 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.348937 27472 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:55.409425 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 364) took 1251 ms. Trace:
I20240426 13:19:55.398867 27495 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.411494 25027 rpcz_store.cc:276] 0426 13:19:54.158004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.272365 (+1114361us) service_pool.cc:225] Handling call
0426 13:19:55.272741 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.272986 (+   245us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:19:55.273336 (+   350us) log.cc:842] Serialized 14 byte log entry
0426 13:19:55.409096 (+135760us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.409147 (+    51us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:55.409194 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.409258 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.409374 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":170,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":25655,"spinlock_wait_cycles":135936,"thread_start_us":109901,"threads_started":1}
I20240426 13:19:55.418892 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:55.421945 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65620.9us
I20240426 13:19:55.422645 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:55.429930 24954 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:55.350363 27677 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
W20240426 13:19:55.434264 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61914.2us
I20240426 13:19:55.341925 27647 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.428998 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:55.429250 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:55.435096 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:55.431876 25027 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:19:55.412098 27379 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.438040 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:55.412786 27472 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:55.455802 25027 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:55.458063 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.416728 27685 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.464605 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:55.423089 27526 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.473881 25101 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:55.343267 27687 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.412529 27486 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20240426 13:19:55.471601 25027 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.474293 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.356618 27710 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.492043 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58769.3us
I20240426 13:19:55.495756 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:55.498486 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:55.493773 24954 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:55.499542 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:55.502951 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.456156 27715 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.474439 27472 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:55.526899 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:55.502712 27712 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20240426 13:19:55.490116 27682 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:55.494705 27303 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.470934 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20240426 13:19:55.420372 27740 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.492596 27482 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.519575 27416 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.519856 27738 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:55.545974 27684 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.524693 27474 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.494421 27526 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.568244 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98116.8us
I20240426 13:19:55.548002 27715 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:55.550153 27725 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.568812 27738 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.603042 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1631) took 1659 ms. Trace:
I20240426 13:19:55.602892 27738 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.604120 27738 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:55.570848 27416 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:19:55.579033 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66426.7us
I20240426 13:19:55.559818 27742 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.437130 27697 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.584245 27486 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.580127 27749 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.582154 27735 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:55.579762 27739 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 13:19:55.585393 27745 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
I20240426 13:19:55.581543 27757 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20240426 13:19:55.582593 27765 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:55.570082 27482 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.594659 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.593365 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 445) took 2180 ms. Trace:
W20240426 13:19:55.598778 27744 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
W20240426 13:19:55.600852 27107 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.555s	user 0.015s	sys 0.000s
I20240426 13:19:55.601554 27725 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.588376 27474 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.605583 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 106087us
I20240426 13:19:55.603516 25105 rpcz_store.cc:276] 0426 13:19:53.943151 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.943726 (+   575us) service_pool.cc:225] Handling call
0426 13:19:53.944053 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:54.000048 (+ 55995us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.602768 (+1602720us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b54003e12d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:55.602951 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127107200}
I20240426 13:19:55.617273 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:55.605624 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.538s (SENT) [suppressed 2 similar messages]
I20240426 13:19:55.586591 27687 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.612515 27758 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20240426 13:19:55.628764 27773 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:55.490259 27379 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.625531 27742 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.626885 27749 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.634083 27765 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:55.640812 27740 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.640385 27771 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.639518 27759 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:19:55.635782 27734 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.446s	user 0.003s	sys 0.000s
W20240426 13:19:55.625881 27676 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.392s	user 0.015s	sys 0.006s
I20240426 13:19:55.638181 25617 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:55.629428 27735 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.647042 27647 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.639295 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 561) took 1330 ms. Trace:
I20240426 13:19:55.641825 27715 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:55.645387 27416 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.640636 27770 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:55.650357 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:55.651708 24952 rpcz_store.cc:276] 0426 13:19:53.413350 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.413564 (+   214us) service_pool.cc:225] Handling call
0426 13:19:53.413919 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:53.490069 (+ 76150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:53.490098 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:53.490138 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:53.490167 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.591886 (+2101719us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:55.593193 (+  1307us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.593309 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174725376}
I20240426 13:19:55.564507 27495 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.656891 27107 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.657337 27482 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.659078 27350 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:55.661873 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 360) took 1640 ms. Trace:
W20240426 13:19:55.647572 27666 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.007s	sys 0.002s
I20240426 13:19:55.677635 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.627534 27486 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.647346 27651 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.651535 27730 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20240426 13:19:55.679694 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.960275 27742 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.962039 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1448) took 3465 ms (client timeout 1788 ms). Trace:
W20240426 13:19:55.962852 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1580) took 1753 ms (client timeout 1538 ms). Trace:
W20240426 13:19:55.965534 27310 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.921s	user 0.008s	sys 0.002s
I20240426 13:19:55.966531 27486 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.972887 27526 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.975095 27474 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.976300 27696 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.956s	user 0.011s	sys 0.005s
I20240426 13:19:55.977473 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:55.755486 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 234669us
I20240426 13:19:55.993907 25029 rpcz_store.cc:276] 0426 13:19:54.021674 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.577809 (+556135us) service_pool.cc:225] Handling call
0426 13:19:54.609019 (+ 31210us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.609229 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.609256 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.620417 (+ 11161us) log.cc:842] Serialized 26 byte log entry
0426 13:19:54.998157 (+377740us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:54.998204 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.998288 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.661591 (+663303us) raft_consensus.cc:1671] finished
0426 13:19:55.661619 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.661804 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":306647,"threads_started":1,"wal-append.queue_time_us":619015}
W20240426 13:19:55.990417 25103 rpcz_store.cc:269] 0426 13:19:52.496864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.496980 (+   116us) service_pool.cc:225] Handling call
0426 13:19:55.961751 (+3464771us) spinlock_profiling.cc:243] Waited 2.75 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.961937 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6324418432}
W20240426 13:19:55.991140 24948 rpcz_store.cc:269] 0426 13:19:54.209870 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.210094 (+   224us) service_pool.cc:225] Handling call
0426 13:19:55.962675 (+1752581us) spinlock_profiling.cc:243] Waited 90.2 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.962770 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207499008}
W20240426 13:19:55.954639 27510 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.597s	user 0.008s	sys 0.000s
W20240426 13:19:55.989504 27666 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.011s	sys 0.006s
I20240426 13:19:55.727453 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.685314 25099 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:56.100690 25031 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:19:56.101153 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58967.5us
I20240426 13:19:56.102267 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.102587 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:56.103369 25029 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.041924 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.030839 25623 rpcz_store.cc:276] 0426 13:19:54.309089 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.309337 (+   248us) service_pool.cc:225] Handling call
0426 13:19:55.639081 (+1329744us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.639217 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":928334208}
I20240426 13:19:56.009349 27767 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.146291 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1992) took 2208 ms (client timeout 2208 ms). Trace:
I20240426 13:19:55.954456 27379 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.989758 27716 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.010042 27725 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:56.146705 25023 rpcz_store.cc:269] 0426 13:19:53.937587 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.950636 (+ 13049us) service_pool.cc:225] Handling call
0426 13:19:56.146074 (+2195438us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.146224 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139899776}
W20240426 13:19:56.134904 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 529281us
I20240426 13:19:56.163630 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.164443 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:56.164705 25105 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:56.166113 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.148262 25022 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:56.066135 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:56.008350 27482 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.153712 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.157402 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.173820 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.173945 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:55.922618 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:56.174417 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:56.007575 27310 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.989946 27765 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:56.097764 27656 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.203s	user 0.008s	sys 0.003s
I20240426 13:19:56.168133 25101 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:56.101233 27755 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.727545 27637 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.174247 25103 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:56.101428 27487 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:56.039369 27770 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:56.167630 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101629us
W20240426 13:19:56.193763 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.043s	sys 0.078s
W20240426 13:19:56.158802 27738 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.554s	user 0.009s	sys 0.000s
I20240426 13:19:55.735790 27438 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.457s
I20240426 13:19:55.734158 27773 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:56.148101 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 391) took 1917 ms. Trace:
I20240426 13:19:56.178014 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.178354 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.183223 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.184379 25100 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:56.145838 27416 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.145941 27739 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.189627 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.192096 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 465) took 1469 ms. Trace:
I20240426 13:19:56.234206 24956 rpcz_store.cc:276] 0426 13:19:54.722866 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.723033 (+   167us) service_pool.cc:225] Handling call
0426 13:19:54.764567 (+ 41534us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.793819 (+ 29252us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.793865 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.799457 (+  5592us) log.cc:842] Serialized 26 byte log entry
0426 13:19:55.015787 (+216330us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:55.015832 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.015906 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:56.191816 (+1175910us) raft_consensus.cc:1671] finished
0426 13:19:56.191844 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.192008 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21934,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6187,"thread_start_us":143251,"threads_started":1,"wal-append.queue_time_us":347797}
I20240426 13:19:56.238766 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.148804 27759 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.842306 27710 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.098100 27677 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.230824 25023 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:56.248054 27677 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.176955 25029 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.249920 27773 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.250790 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:56.194097 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.043s	sys 0.078s
I20240426 13:19:56.205857 27755 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.204885 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:56.175554 27715 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.253324 27715 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.254040 27749 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.188616 27697 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.052s	sys 0.087s
W20240426 13:19:56.254983 27697 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.052s	sys 0.088s
I20240426 13:19:56.193645 27487 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:56.256904 27487 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.257820 27651 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.006201 27771 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.258224 27416 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.225796 24952 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:56.225497 25617 rpcz_store.cc:276] 0426 13:19:54.194709 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.204497 (+  9788us) service_pool.cc:225] Handling call
0426 13:19:55.637784 (+1433287us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:55.637999 (+   215us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.005965 (+367966us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.112009 (+106044us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54551808}
I20240426 13:19:56.188639 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:19:56.251840 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 637046us
I20240426 13:19:56.178263 27482 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.161828 27740 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.156505 27767 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.189121 27735 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.166424 27716 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.351769 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.022s	sys 0.045s
W20240426 13:19:56.357112 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.022s	sys 0.046s
I20240426 13:19:56.337257 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 366) took 1661 ms. Trace:
I20240426 13:19:56.338034 24952 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:56.200712 27770 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.358119 25028 rpcz_store.cc:276] 0426 13:19:54.676256 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.425812 (+749556us) service_pool.cc:225] Handling call
0426 13:19:55.447608 (+ 21796us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.595545 (+147937us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.595596 (+    51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.595647 (+    51us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:55.595687 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.300287 (+704600us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003af2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:56.337073 (+ 36786us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.337198 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":292663168}
I20240426 13:19:56.270215 24948 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:56.369401 24952 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:56.375573 24952 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 13:19:56.375931 24952 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.378198 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.378871 24948 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.370869 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:56.336192 27698 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.887s	user 0.009s	sys 0.005s
I20240426 13:19:56.368913 27331 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.372361 27771 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.372293 27310 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.373039 27730 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.371093 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.381381 25029 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:56.266337 24954 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:19:56.098434 27676 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.397176 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2006) took 1608 ms. Trace:
I20240426 13:19:56.397545 25023 rpcz_store.cc:276] 0426 13:19:54.788856 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.396393 (+1607537us) service_pool.cc:225] Handling call
0426 13:19:56.396691 (+   298us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.396862 (+   171us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:56.396888 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.396928 (+    40us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:56.396956 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.397014 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.397122 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.398442 25029 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:56.262395 27438 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:56.260752 27416 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.202333 27656 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.399037 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2005) took 1614 ms. Trace:
I20240426 13:19:56.408140 25023 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:56.408771 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2007) took 1573 ms. Trace:
I20240426 13:19:56.408830 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.369400 27774 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.262073 27778 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.409198 25023 rpcz_store.cc:276] 0426 13:19:54.835084 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.404626 (+1569542us) service_pool.cc:225] Handling call
0426 13:19:56.404947 (+   321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.408559 (+  3612us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.408718 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.407608 25029 rpcz_store.cc:276] 0426 13:19:54.785158 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.381968 (+1596810us) service_pool.cc:225] Handling call
0426 13:19:56.398154 (+ 16186us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.398807 (+   653us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.398988 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.411504 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.360194 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:56.412441 25025 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.335330 27107 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.423275 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:56.403012 27749 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.398514 27331 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.425801 25102 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:56.204960 27637 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.414498 27416 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20240426 13:19:56.481016 27771 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20240426 13:19:56.488283 27749 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:56.489794 27676 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.263543 27739 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.475551 27656 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.473624 26996 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.491835 27739 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.400249 27310 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.492565 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.044s	sys 0.053s
W20240426 13:19:56.492887 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.044s	sys 0.053s
I20240426 13:19:56.493661 27310 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.494180 27310 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.494791 27310 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:56.482314 27673 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.485783 25101 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:56.377403 27740 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.480249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62815.2us
I20240426 13:19:56.467909 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.502378 24952 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:56.430390 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.399741 27647 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.416910 27677 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.491747 27749 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.495747 27755 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.425308 27107 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.526538 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 115110us
I20240426 13:19:56.406486 27697 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20240426 13:19:56.407086 27730 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.528096 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.520700 27715 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.542371 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 305373us
I20240426 13:19:56.546119 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.546592 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:56.550915 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 384865us
I20240426 13:19:56.551720 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:56.552398 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 427804us
I20240426 13:19:56.425904 27774 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.558821 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.520362 27647 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.562675 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.566843 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.567485 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:56.567601 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.417542 27778 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.544431 27749 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:56.561586 27472 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.995s	user 0.011s	sys 0.000s
I20240426 13:19:56.518168 27673 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.547329 27438 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:56.563835 27697 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.530869 27107 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.561774 27755 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.557786 27418 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:56.578013 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.001s	sys 0.022s
I20240426 13:19:56.566792 27310 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:56.582564 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.589378 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:56.583048 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 81152us
I20240426 13:19:56.584118 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:56.585805 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.001s	sys 0.022s
I20240426 13:19:56.592944 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.591781 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:19:56.596387 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.596822 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:56.499380 27759 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.603184 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.608004 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.611196 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.611850 24948 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:56.611745 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:56.613902 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45479.3us
I20240426 13:19:56.612412 24952 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:56.617942 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.618243 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.618826 24952 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.581107 27472 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.585889 27379 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.625499 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.634800 24948 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:56.634925 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.582731 27482 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.627470 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.589802 27793 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.636409 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.638913 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.608325 27438 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:56.605800 27677 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:56.646260 24952 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.647894 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.648473 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.648869 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.649405 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2008) took 1637 ms. Trace:
I20240426 13:19:56.652207 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.650815 25023 rpcz_store.cc:276] 0426 13:19:55.012130 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.418429 (+1406299us) service_pool.cc:225] Handling call
0426 13:19:56.418733 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.419068 (+   335us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:56.579767 (+160699us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.649100 (+ 69333us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.649153 (+    53us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:56.649197 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.649260 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.649358 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":216709,"thread_start_us":152816,"threads_started":2}
I20240426 13:19:56.654636 25620 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:56.650033 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.578732 27685 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20240426 13:19:56.625456 27716 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.655735 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.658926 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.572669 27788 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20240426 13:19:56.590518 27755 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.662053 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.662860 24948 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:56.594861 27418 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.662987 25105 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:56.664616 25023 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:56.665619 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.667660 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:56.666544 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.605572 27749 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 13:19:56.671638 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:56.609616 27676 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:56.672863 25023 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:56.661093 27783 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20240426 13:19:56.657641 27482 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:56.647315 27438 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.680685 27796 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:56.681011 27438 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:56.681337 27796 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.681835 27796 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:56.680735 27482 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.685137 27482 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.657687 27715 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:56.679256 27755 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.688392 25023 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:19:56.656806 27495 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:56.651237 27677 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.689493 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2010) took 1325 ms. Trace:
I20240426 13:19:56.691483 25023 rpcz_store.cc:276] 0426 13:19:55.364579 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.687683 (+1323104us) service_pool.cc:225] Handling call
0426 13:19:56.687976 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.689155 (+  1179us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.689429 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.615381 27794 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:56.658164 27767 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.675721 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.711900 27801 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:56.712474 27801 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:56.715618 27786 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20240426 13:19:56.716826 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 476) took 1051 ms. Trace:
I20240426 13:19:56.717151 27482 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.717209 24947 rpcz_store.cc:276] 0426 13:19:55.665605 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.665776 (+   171us) service_pool.cc:225] Handling call
0426 13:19:55.667018 (+  1242us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.667340 (+   322us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.667370 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.674216 (+  6846us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.357685 (+683469us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:56.357740 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.357833 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:56.716599 (+358766us) raft_consensus.cc:1671] finished
0426 13:19:56.716618 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.716770 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57693,"raft.queue_time_us":939,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":45750,"thread_start_us":583209,"threads_started":1,"wal-append.queue_time_us":836096}
I20240426 13:19:56.718866 27451 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:56.719174 24955 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20240426 13:19:56.719326 27482 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.721109 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1589) took 1407 ms. Trace:
I20240426 13:19:56.721483 24953 rpcz_store.cc:276] 0426 13:19:55.313258 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.314100 (+   842us) service_pool.cc:225] Handling call
0426 13:19:56.720904 (+1406804us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b54002ca6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.721050 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14568832}
I20240426 13:19:56.722667 27774 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.724529 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1987) took 2798 ms (client timeout 1686 ms). Trace:
W20240426 13:19:56.724999 25024 rpcz_store.cc:269] 0426 13:19:53.925774 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.927632 (+  1858us) service_pool.cc:225] Handling call
0426 13:19:56.724316 (+2796684us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.724460 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1404053760}
I20240426 13:19:56.713318 27801 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.728888 25027 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.734781 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2011) took 1315 ms. Trace:
I20240426 13:19:56.735298 25023 rpcz_store.cc:276] 0426 13:19:55.419443 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.712105 (+1292662us) service_pool.cc:225] Handling call
0426 13:19:56.712401 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.734330 (+ 21929us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:56.734375 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.734433 (+    58us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:56.734479 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.734571 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.734714 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21571}
I20240426 13:19:56.664999 27740 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.668720 27716 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:56.648203 27379 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.742421 25024 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:19:56.747545 27495 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.747720 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.748013 27774 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.751034 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 154311us
I20240426 13:19:56.752461 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.754413 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 370) took 1317 ms. Trace:
I20240426 13:19:56.677505 27637 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.754920 25024 rpcz_store.cc:276] 0426 13:19:55.437179 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.725784 (+1288605us) service_pool.cc:225] Handling call
0426 13:19:56.726139 (+   355us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.754093 (+ 27954us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.754344 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.694259 27677 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:56.758625 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:56.759581 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:56.761152 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.764707 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.766722 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.768474 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.748845 27716 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.787951 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.785347 27767 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:56.790035 27767 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.790513 27767 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:56.790287 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.749830 27730 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.796680 27794 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.784732 27637 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.798292 27637 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.799154 27331 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:56.801381 27732 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.660s	user 0.006s	sys 0.000s
I20240426 13:19:56.797363 27794 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.817004 27794 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.817705 27637 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.818904 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:56.821118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61607.4us
I20240426 13:19:56.803225 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.775779 25027 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20240426 13:19:56.821213 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:56.820539 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 193144us
I20240426 13:19:56.822577 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.814787 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:56.823334 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1454) took 3548 ms (client timeout 1737 ms). Trace:
I20240426 13:19:56.823740 25105 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.825492 25027 pending_rounds.cc:76] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 7
I20240426 13:19:56.826046 25027 pending_rounds.cc:97] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20240426 13:19:56.826427 25027 raft_consensus.cc:2866] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.761508 27647 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.651239 27793 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.828598 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.789218 27716 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:56.828871 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.801712 27331 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.849715 24947 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 9
W20240426 13:19:56.823740 25096 rpcz_store.cc:269] 0426 13:19:53.275058 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.308212 (+ 33154us) service_pool.cc:225] Handling call
0426 13:19:56.823174 (+3514962us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.823268 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132486912}
I20240426 13:19:56.850483 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.851310 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.860632 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.862403 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 367) took 2181 ms. Trace:
I20240426 13:19:56.822499 27746 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.757156 27495 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.862808 25027 rpcz_store.cc:276] 0426 13:19:54.678726 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.431491 (+752765us) service_pool.cc:225] Handling call
0426 13:19:55.431761 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.728537 (+1296776us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b58004c01b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:19:56.827181 (+ 98644us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.860007 (+ 32826us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":385102720}
I20240426 13:19:56.873472 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.761114 27774 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.877619 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.878350 25027 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:56.881415 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:56.829407 27150 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:56.883865 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:56.885970 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.888069 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.887858 25027 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:19:56.834790 27685 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:56.852589 27673 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.876269 27778 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.875185 27472 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.891101 27150 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:56.892740 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 300939us
I20240426 13:19:56.891671 27804 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.895287 27685 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.919744 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.933243 25615 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:56.910650 27799 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.905288 27773 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.906555 27647 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:56.904649 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.891155 27808 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:56.928084 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54723.3us
W20240426 13:19:56.920060 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 151668us
I20240426 13:19:56.935056 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 371) took 1372 ms. Trace:
I20240426 13:19:56.963480 25024 rpcz_store.cc:276] 0426 13:19:55.562486 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.755936 (+1193450us) service_pool.cc:225] Handling call
0426 13:19:56.786497 (+ 30561us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.786774 (+   277us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:19:56.787117 (+   343us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.934734 (+147617us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.934797 (+    63us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:56.934841 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.934904 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.935006 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13269,"prepare.queue_time_us":943,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":13373,"thread_start_us":133967,"threads_started":1}
I20240426 13:19:56.964331 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.866315 27107 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.949254 25096 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 13:19:56.965473 25096 pending_rounds.cc:76] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 1
I20240426 13:19:56.965675 25096 pending_rounds.cc:97] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 13:19:56.965905 25096 raft_consensus.cc:2866] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.914988 27794 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.919338 27498 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.885427 27746 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.940122 27482 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:56.934302 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.960997 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.959098 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 369) took 1698 ms. Trace:
I20240426 13:19:56.965252 25105 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:56.967806 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.926240 27685 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.919179 27150 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:56.965957 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.977523 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.950248 27799 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.948359 27418 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.999080 24949 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.979702 25029 rpcz_store.cc:276] 0426 13:19:55.261131 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.412057 (+1150926us) service_pool.cc:225] Handling call
0426 13:19:56.428138 (+ 16081us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.428338 (+   200us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:56.901577 (+473239us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.958751 (+ 57174us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.958802 (+    51us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:56.958845 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.958916 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.959043 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67194,"prepare.queue_time_us":243934,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":3616,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":8292,"thread_start_us":462403,"threads_started":2}
I20240426 13:19:56.955296 27327 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.978813 27482 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.009547 27482 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:57.002225 25101 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.002955 27495 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.004022 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.004334 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.982739 27730 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:56.968984 27647 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.017172 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:57.002506 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:57.020896 25029 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:57.023641 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57790.3us
I20240426 13:19:57.025341 24949 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:56.975112 27755 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
W20240426 13:19:56.982452 27797 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.271s	user 0.001s	sys 0.001s
I20240426 13:19:57.029666 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:57.034332 27770 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.007s
I20240426 13:19:57.041492 27107 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.041936 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.957628 27773 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.026708 27740 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:56.962179 27801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 13:19:57.020870 27379 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.058553 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:56.974876 27794 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.061496 27793 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.061944 27810 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:57.062443 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:57.063095 27773 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.053755 27496 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.080893 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.062875 27482 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:57.063532 27812 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20240426 13:19:57.063824 25097 raft_consensus.cc:595] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:57.111644 25097 raft_consensus.cc:603] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:57.111889 25029 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:57.055938 27740 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.934737 27804 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:57.090479 27821 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20240426 13:19:57.090585 27799 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:57.105165 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46693us
I20240426 13:19:56.978598 27808 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.064733 27735 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20240426 13:19:57.116633 27709 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:57.050565 27770 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.065096 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:19:57.064237 27816 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20240426 13:19:57.064064 27438 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.100458 27716 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.110101 25105 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.108044 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1586) took 1797 ms. Trace:
I20240426 13:19:57.113818 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1641) took 1770 ms. Trace:
I20240426 13:19:57.131466 25099 rpcz_store.cc:276] 0426 13:19:55.343728 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.343977 (+   249us) service_pool.cc:225] Handling call
0426 13:19:55.358079 (+ 14102us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.099646 (+741567us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.113597 (+1013951us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400168d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.113765 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":737609088}
I20240426 13:19:57.135457 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:57.137032 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72006.3us
I20240426 13:19:57.138540 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:57.115521 27740 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.156410 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.157114 25099 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:57.130358 24951 rpcz_store.cc:276] 0426 13:19:55.310711 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.311265 (+   554us) service_pool.cc:225] Handling call
0426 13:19:55.338687 (+ 27422us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.339058 (+   371us) raft_consensus.cc:1520] Early marking committed up to index 1
0426 13:19:55.339096 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.339500 (+   404us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.728025 (+1388525us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b1800184960. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:56.904935 (+176910us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:56.904972 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.905036 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.105319 (+200283us) raft_consensus.cc:1671] finished
0426 13:19:57.105338 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.107969 (+  2631us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6008,"raft.queue_time_us":2859,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11973,"spinlock_wait_cycles":4834560,"thread_start_us":147035,"threads_started":1,"wal-append.queue_time_us":1704021}
I20240426 13:19:57.159066 27793 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.158931 27796 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.159675 27793 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.161237 27716 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.161327 27438 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.161692 27793 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.161904 27716 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.111433 27496 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.162317 27810 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:57.162153 27685 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.164009 27496 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.162326 27716 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.164764 27716 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.164956 27379 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.165472 27379 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.167405 27379 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.114894 27768 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:57.107758 27673 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:57.170159 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 467) took 3151 ms. Trace:
I20240426 13:19:57.172590 25023 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:57.178561 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:57.174032 25098 rpcz_store.cc:276] 0426 13:19:54.018851 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.018969 (+   118us) service_pool.cc:225] Handling call
0426 13:19:54.019330 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.173952 (+154622us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.173996 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.201717 (+ 27721us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.414223 (+2212506us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b1800437c80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:56.540766 (+126543us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:56.540829 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.540925 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.117878 (+576953us) raft_consensus.cc:1671] finished
0426 13:19:57.117902 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.170086 (+ 52184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164034,"raft.queue_time_us":172246,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9526,"spinlock_wait_cycles":18676480,"thread_start_us":126308,"threads_started":1,"wal-append.queue_time_us":2389752}
W20240426 13:19:57.121651 27823 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20240426 13:19:57.186667 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:57.163425 27810 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.189983 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.191205 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.131165 27774 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.155910 27482 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.066164 27755 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:57.173033 27735 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:19:57.202520 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 160678us
I20240426 13:19:57.216606 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.203120 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.173544 27673 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.120465 27709 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.217972 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.219957 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.190407 27801 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.189986 27685 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.182982 27677 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:57.226111 27673 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:57.179289 27773 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20240426 13:19:57.223529 27482 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.227807 27801 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:57.294616 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74838.6us
I20240426 13:19:57.321404 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.194591 27793 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240426 13:19:57.226325 27735 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:57.312343 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79028.9us
W20240426 13:19:57.313556 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84698.6us
I20240426 13:19:57.315322 25101 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.219534 27628 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:57.328305 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.329434 25098 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:57.176348 27768 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.295677 27496 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.331594 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:19:57.323307 27786 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s	user 0.007s	sys 0.000s
I20240426 13:19:57.318475 27438 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.319977 27482 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.331116 27637 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.322472 27107 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:57.323057 27801 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.324950 27495 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.325134 27773 consensus_queue.cc:1225] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:19:57.343684 27496 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.351979 27827 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:57.324851 27150 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.348484 27438 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.334297 27794 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.341933 27804 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.348448 27482 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.369827 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.359082 27107 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.357589 27827 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:57.448237 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126918us
I20240426 13:19:57.449272 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.453835 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.454838 25101 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:57.370018 27796 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.353854 27637 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.375114 27793 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:57.357321 27495 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.394219 27770 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.341733 27755 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.463496 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1633) took 3386 ms. Trace:
I20240426 13:19:57.501106 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.439777 27808 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.400799 27827 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:57.650285 24952 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:19:57.632629 25104 rpcz_store.cc:276] 0426 13:19:54.077142 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.078167 (+  1025us) service_pool.cc:225] Handling call
0426 13:19:54.086311 (+  8144us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.191143 (+104832us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.191170 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.191560 (+   390us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.974425 (+2782865us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b18003f61c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:57.094149 (+119724us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:57.094187 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.094270 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.456236 (+361966us) raft_consensus.cc:1671] finished
0426 13:19:57.456258 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.463427 (+  7169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92698,"raft.queue_time_us":118132,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":8366,"spinlock_wait_cycles":60471680,"thread_start_us":119570,"threads_started":1,"wal-append.queue_time_us":3031559}
I20240426 13:19:57.623661 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20240426 13:19:57.652288 25030 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:19:57.652585 25030 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240426 13:19:57.652834 25030 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:57.654732 27495 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.656783 24956 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20240426 13:19:57.658828 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 472250us
I20240426 13:19:57.653504 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 362) took 3628 ms. Trace:
I20240426 13:19:57.497893 27843 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.439742 27107 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:57.645169 27765 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.437s	user 0.012s	sys 0.009s
I20240426 13:19:57.674903 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:57.677938 27716 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.000s
I20240426 13:19:57.678678 27832 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.679215 27832 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.679709 27839 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.321s
I20240426 13:19:57.656551 27474 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.680529 27474 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.456738 27740 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.665869 25101 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:57.667181 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 215783us
I20240426 13:19:57.661149 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.678385 27716 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.659976 27677 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:19:57.681241 27834 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20240426 13:19:57.691848 27767 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.858s	user 0.010s	sys 0.001s
I20240426 13:19:57.698031 27150 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.399052 27735 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.695349 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.701673 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.702055 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.702306 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.703220 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.703999 24956 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:57.705691 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.706235 24953 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:57.709046 27637 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.713618 27835 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:57.713912 27716 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.714749 27847 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:57.716300 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:57.716475 27350 raft_consensus.cc:1054] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Unable to promote non-voter 1590f80bdade41c6a755b8501c55fe16: Illegal state: Replica 1e87e7bb414642aeb72fe0d2ff72cc2e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:57.716921 24951 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:57.718700 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.719345 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.721264 24953 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:57.747113 27831 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20240426 13:19:57.748477 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.749217 24951 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:57.754770 27740 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.683754 27765 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.721815 27438 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.737633 27801 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.761844 27832 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.763027 27801 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.671089 27843 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.741389 27150 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.666532 25030 rpcz_store.cc:276] 0426 13:19:54.024802 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.692278 (+667476us) service_pool.cc:225] Handling call
0426 13:19:54.717392 (+ 25114us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.622953 (+2905561us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b540006c0a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda9031a057 00007fda9031a740 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:19:57.653254 (+ 30301us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.653452 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18504448}
I20240426 13:19:57.751217 27637 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.718681 27767 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.777541 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:57.773165 27150 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.782680 27150 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.779561 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.789528 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:57.791334 27677 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.801640 24951 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:57.791832 27677 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.804901 25105 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:57.805899 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 147957us
I20240426 13:19:57.796166 27416 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.773418 27637 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.800303 27832 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.811033 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
W20240426 13:19:57.810635 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146036us
I20240426 13:19:57.821491 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.824216 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.829442 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.830273 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.831228 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.850503 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:57.807428 27107 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.808517 27843 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.800364 27673 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.888864 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.882293 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.906994 25096 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:57.890383 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.909507 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:57.912961 27827 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.916635 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 605) took 2477 ms. Trace:
I20240426 13:19:57.879839 27832 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.916746 27827 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.924188 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.925444 24953 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.917263 25620 rpcz_store.cc:276] 0426 13:19:55.438951 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.439178 (+   227us) service_pool.cc:225] Handling call
0426 13:19:55.471415 (+ 32237us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.661579 (+1190164us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.916349 (+1254770us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.916551 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2720682880}
W20240426 13:19:57.841256 27759 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.010s	sys 0.003s
I20240426 13:19:57.914356 27496 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.933573 24948 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.934676 24951 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:19:57.937506 27640 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:57.937734 27677 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.935798 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1491) took 2311 ms. Trace:
I20240426 13:19:57.943029 25027 rpcz_store.cc:276] 0426 13:19:55.624598 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.881606 (+1257008us) service_pool.cc:225] Handling call
0426 13:19:56.883269 (+  1663us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.892953 (+  9684us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.935575 (+1042622us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.935740 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9804544}
I20240426 13:19:57.944242 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.944855 25027 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:57.944904 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1598) took 1746 ms. Trace:
I20240426 13:19:57.945719 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.948668 27640 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.949071 27640 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:57.950316 27795 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 1.361s	user 0.011s	sys 0.000s
I20240426 13:19:57.941031 24949 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:57.953081 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.953523 27637 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.953790 25027 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:57.954120 27765 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.954798 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.955226 27767 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.955564 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:57.956547 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.957407 27673 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.957912 27709 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.958102 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.959710 25027 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.945398 24955 rpcz_store.cc:276] 0426 13:19:56.198388 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.198531 (+   143us) service_pool.cc:225] Handling call
0426 13:19:56.718914 (+520383us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.719575 (+   661us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.944673 (+1225098us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540004bd50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:57.944835 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195890048}
I20240426 13:19:57.963639 24947 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.963953 24949 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:57.959900 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.966658 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.967263 24948 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:57.941715 27759 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.011s	sys 0.003s
I20240426 13:19:57.969039 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 401) took 2539 ms. Trace:
I20240426 13:19:57.969550 25616 rpcz_store.cc:276] 0426 13:19:55.429160 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.433601 (+  4441us) service_pool.cc:225] Handling call
0426 13:19:55.454141 (+ 20540us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.787164 (+333023us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:19:55.834023 (+ 46859us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:56.106725 (+272702us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.519065 (+412340us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:56.521715 (+  2650us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.526781 (+  5066us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.949691 (+1422910us) spinlock_profiling.cc:243] Waited 8.52 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:57.961706 (+ 12015us) raft_consensus.cc:1671] finished
0426 13:19:57.963352 (+  1646us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.968974 (+  5622us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102169,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3997,"spinlock_wait_cycles":19588864,"thread_start_us":512199,"threads_started":2,"wal-append.queue_time_us":379994}
I20240426 13:19:57.968216 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.980139 24948 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:19:57.837769 27416 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.961165 27843 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.986477 24953 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:57.986609 24949 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20240426 13:19:57.954746 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 399) took 2526 ms (client timeout 2021 ms). Trace:
W20240426 13:19:57.984290 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.001s	sys 0.042s
W20240426 13:19:57.987879 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.002s	sys 0.042s
I20240426 13:19:57.984990 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.966363 27677 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:57.996596 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.997387 24949 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:57.992247 25027 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.992355 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.000552 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:58.001008 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.001858 24949 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:58.004763 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.994761 27677 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.010373 27677 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.012326 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.012876 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.995496 27832 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:57.990471 25618 rpcz_store.cc:269] 0426 13:19:55.428529 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.428681 (+   152us) service_pool.cc:225] Handling call
0426 13:19:57.954493 (+2525812us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:57.954628 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":927631744}
I20240426 13:19:58.009711 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.013062 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:58.018164 24947 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:58.020023 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.022825 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.007678 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:19:58.023962 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2015) took 1871 ms. Trace:
I20240426 13:19:58.024389 25027 rpcz_store.cc:276] 0426 13:19:56.152438 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.007019 (+1854581us) service_pool.cc:225] Handling call
0426 13:19:58.007301 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.023600 (+ 16299us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.023898 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:58.025506 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.995784 27496 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20240426 13:19:58.030336 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.030603 27496 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.031150 27496 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:58.032205 27496 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.038313 27640 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.041118 27416 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:58.044149 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 397) took 3241 ms (client timeout 1746 ms). Trace:
I20240426 13:19:58.044391 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:58.044659 25622 rpcz_store.cc:269] 0426 13:19:54.802955 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.803102 (+   147us) service_pool.cc:225] Handling call
0426 13:19:58.043959 (+3240857us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:58.044071 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4953170816}
I20240426 13:19:58.045025 24951 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.046422 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.043990 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.049237 24955 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:58.049461 27750 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.691s	user 0.011s	sys 0.022s
I20240426 13:19:58.050223 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.050938 24953 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:58.051826 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.054018 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.055537 24949 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:58.056979 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.061259 27496 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.063844 27687 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:58.064657 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.000151 27843 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:58.078824 27728 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20240426 13:19:58.080447 27640 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:58.086089 27628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.008s	sys 0.033s
W20240426 13:19:58.144933 27482 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.770s	user 0.011s	sys 0.000s
I20240426 13:19:58.145485 27482 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.146883 25027 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:58.143124 27677 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:58.153053 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.013s	sys 0.049s
W20240426 13:19:58.153904 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.013s	sys 0.049s
I20240426 13:19:58.151746 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2001) took 3816 ms. Trace:
I20240426 13:19:58.158264 25026 rpcz_store.cc:276] 0426 13:19:54.335531 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.307940 (+972409us) service_pool.cc:225] Handling call
0426 13:19:55.344272 (+ 36332us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.409880 (+ 65608us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.409949 (+    69us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.410029 (+    80us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:55.410094 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.151419 (+2741325us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:58.151483 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.151652 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48932096}
W20240426 13:19:58.126669 27628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.008s	sys 0.034s
I20240426 13:19:58.147934 27482 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.185106 27628 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:58.225399 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.299743 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.226951 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:58.294690 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.008s	sys 0.067s
W20240426 13:19:58.185240 25024 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.080s	user 0.020s	sys 0.056s
W20240426 13:19:58.210366 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.012s	sys 0.012s
W20240426 13:19:58.304235 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.016s	sys 0.009s
I20240426 13:19:58.307438 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.329448 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:58.301226 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.308770 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:58.276484 27683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.483s	user 0.018s	sys 0.030s
I20240426 13:19:58.315702 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.336350 27796 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:58.330821 25030 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:58.299551 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.345538 25026 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:58.339488 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.133s	user 0.008s	sys 0.027s
I20240426 13:19:58.339661 27628 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.319447 27474 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:58.331251 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.017s	sys 0.013s
W20240426 13:19:58.419620 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 759871us
I20240426 13:19:58.332204 25104 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:58.331140 25024 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.226s	user 0.020s	sys 0.056s
I20240426 13:19:58.330716 24951 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:58.330999 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.009s	sys 0.067s
I20240426 13:19:58.426710 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2013) took 2383 ms. Trace:
W20240426 13:19:58.334276 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.018s	sys 0.009s
W20240426 13:19:58.347671 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.142s	user 0.009s	sys 0.027s
W20240426 13:19:58.435634 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 638337us
I20240426 13:19:58.343686 25096 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:19:58.352643 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52009us
I20240426 13:19:58.359148 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.583711 25098 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.584636 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:58.589020 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 152291us
W20240426 13:19:58.330449 27740 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.013s	sys 0.015s
I20240426 13:19:58.595176 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.427223 25024 rpcz_store.cc:276] 0426 13:19:56.043319 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.964933 (+921614us) service_pool.cc:225] Handling call
0426 13:19:56.980847 (+ 15914us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.981020 (+   173us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:56.981675 (+   655us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.422790 (+1441115us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:58.426336 (+  3546us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:58.426406 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.426497 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.426645 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":824,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":59933,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":1437180,"threads_started":2,"wal-append.queue_time_us":144298,"wal-append.run_cpu_time_us":1017,"wal-append.run_wall_time_us":1001047}
W20240426 13:19:58.299330 27835 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.022s	sys 0.021s
I20240426 13:19:58.591815 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.586726 27724 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:58.594702 27740 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.013s	sys 0.015s
I20240426 13:19:58.599581 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.594830 27778 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 13:19:58.595677 27833 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.299731 27775 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.595703 25030 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.654168 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.609483 25621 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:58.662966 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.604256 27843 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.605913 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:58.623268 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.608486 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.625651 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.627213 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.619694 25024 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:58.623502 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.007s	sys 0.018s
I20240426 13:19:58.598412 24951 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
W20240426 13:19:58.669801 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63964.6us
I20240426 13:19:58.617389 27850 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:19:58.624593 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:58.599761 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.017s	sys 0.027s
I20240426 13:19:58.641865 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:58.685433 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.018s	sys 0.027s
W20240426 13:19:58.616950 27835 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.022s	sys 0.022s
I20240426 13:19:58.683790 27640 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240426 13:19:58.648634 27775 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:58.617174 27724 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.664856 25104 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:58.585180 27683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.554s	user 0.020s	sys 0.032s
I20240426 13:19:58.675979 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2020) took 2470 ms. Trace:
W20240426 13:19:58.672044 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.007s	sys 0.018s
I20240426 13:19:58.630342 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.682582 25098 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:58.683698 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.670410 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:58.643747 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.683502 27855 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.688165 27474 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.688352 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.688655 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.611028 26824 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240426 13:19:58.615315 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.698782 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.017s	sys 0.022s
W20240426 13:19:58.731770 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.017s	sys 0.023s
W20240426 13:19:58.732206 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 575463us
I20240426 13:19:58.700495 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 482) took 3827 ms. Trace:
I20240426 13:19:58.733007 25102 rpcz_store.cc:276] 0426 13:19:54.873604 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.873775 (+   171us) service_pool.cc:225] Handling call
0426 13:19:54.874116 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.429195 (+1555079us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.700230 (+2271035us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:58.700429 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3568210816}
W20240426 13:19:58.734836 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 119614us
I20240426 13:19:58.735091 25102 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:19:58.632148 26679 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:58.702668 25024 rpcz_store.cc:276] 0426 13:19:56.205676 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.610267 (+2404591us) service_pool.cc:225] Handling call
0426 13:19:58.610581 (+   314us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.675681 (+ 65100us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.675918 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:58.704159 25099 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:58.683048 27677 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.597714 27482 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:58.748756 25098 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.749318 25098 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.719691 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.668000 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.668435 27868 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.735746 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.738175 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.775063 26824 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:58.785712 27863 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:19:58.785782 26824 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.788280 27775 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.789256 27647 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.789592 27775 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.789901 27647 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:58.795162 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.797312 25096 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 13:19:58.804770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69129us
I20240426 13:19:58.699221 27724 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.769795 27766 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.726605 27855 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.777172 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.810731 25024 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.810914 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.812495 27740 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:58.812927 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.814100 27740 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:58.815676 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.763022 27850 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:19:58.762544 27474 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.778188 27778 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.778843 27496 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.778905 27796 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:58.779578 27852 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.780696 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.770925 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.781914 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:58.750264 25098 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.819952 25099 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:58.820370 25098 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.805166 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.795948 25104 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 9
I20240426 13:19:58.806879 26824 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:58.757755 25030 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:58.850081 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.795163 27868 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:58.829025 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47206.5us
I20240426 13:19:58.806541 27862 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:58.818209 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164915us
I20240426 13:19:58.861966 25096 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:58.816819 27855 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.818866 27724 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.867396 27868 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.859072 27862 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:58.864763 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:58.819664 27677 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:58.851583 27850 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:58.816262 27766 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.883656 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.889165 25030 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.893134 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.893708 25030 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:58.897449 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2014) took 2789 ms. Trace:
I20240426 13:19:58.849232 27796 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.897729 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.849011 27496 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.897992 25029 rpcz_store.cc:276] 0426 13:19:56.108521 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.115343 (+1006822us) service_pool.cc:225] Handling call
0426 13:19:57.115647 (+   304us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:57.130996 (+ 15349us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:57.131042 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:57.172318 (+ 41276us) log.cc:842] Serialized 26 byte log entry
0426 13:19:58.435743 (+1263425us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:58.435780 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.435846 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:58.897244 (+461398us) raft_consensus.cc:1671] finished
0426 13:19:58.897263 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.897394 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25561,"raft.queue_time_us":46289,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":1229601,"threads_started":1,"wal-append.queue_time_us":1606684}
I20240426 13:19:58.886907 27850 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.878741 27862 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:58.897459 25069 connection.cc:232] RPC call timeout handler was delayed by -2.0278975679999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:19:58.901185 25030 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:58.857535 27778 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.878629 27677 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:58.886950 27867 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.878579 27883 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:58.817376 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.929307 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 112013us
W20240426 13:19:58.930222 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 145451us
I20240426 13:19:58.931010 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.908304 25069 connection.cc:232] RPC call timeout handler was delayed by -2.0278975679999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:19:58.876827 27699 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20240426 13:19:58.942222 25069 connection.cc:232] RPC call timeout handler was delayed by -1.9963975680000203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20240426 13:19:58.942569 25069 connection.cc:232] RPC call timeout handler was delayed by -1.88029756800006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20240426 13:19:58.915228 25104 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
W20240426 13:19:58.942831 25069 connection.cc:232] RPC call timeout handler was delayed by -1.7308975680000458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20240426 13:19:58.943147 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6982937919999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20240426 13:19:58.943429 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6921975679999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20240426 13:19:58.943706 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6642975680000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20240426 13:19:58.944005 25069 connection.cc:232] RPC call timeout handler was delayed by -1.660697568000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20240426 13:19:58.944299 25069 connection.cc:232] RPC call timeout handler was delayed by -1.660697568000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20240426 13:19:58.944594 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6541937920000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
W20240426 13:19:58.944911 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6327975680000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20240426 13:19:58.945207 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6327975680000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20240426 13:19:58.945493 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6210975679999819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20240426 13:19:58.945780 25069 connection.cc:232] RPC call timeout handler was delayed by -1.554497567999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20240426 13:19:58.946071 25069 connection.cc:232] RPC call timeout handler was delayed by -1.5391975679999632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20240426 13:19:58.946349 25069 connection.cc:232] RPC call timeout handler was delayed by -1.113497567999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25379999999999997s for a response
W20240426 13:19:58.946655 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0864975680000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25680000000000003s for a response
W20240426 13:19:58.946929 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0864975680000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25680000000000003s for a response
I20240426 13:19:58.947504 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.947405 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:58.936946 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.949195 25029 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.949740 25069 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:58.950042 25069 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:58.899757 27376 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.950632 27376 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.952241 26824 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:58.952764 27376 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.953055 26824 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:58.954456 27706 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:58.954874 27706 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.955693 27706 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.958391 25624 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:58.959286 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.959959 25621 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.905213 25098 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:58.961871 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.962761 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.962960 24955 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:58.860810 27852 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.934306 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.961871 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.965152 24952 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:58.958555 27706 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:58.940594 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:58.983572 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.984771 25030 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:19:58.990412 27852 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.990713 27868 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:58.991636 24949 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:58.994741 26824 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:58.996310 27656 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:58.997138 27656 consensus_queue.cc:1225] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:58.997730 27656 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:58.999212 27656 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:59.000375 27640 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.000833 27640 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.995824 26824 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.009927 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.926394 27677 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.009096 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.907343 27850 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.015605 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.925256 27867 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:59.019421 24949 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.019770 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:59.007295 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.021402 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.018456 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.002702 27676 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.001753 27640 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.037238 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.025084 25030 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:59.034924 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94437.6us
I20240426 13:19:59.039819 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.041072 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43677us
I20240426 13:19:59.041441 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:59.043588 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.044014 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:59.016707 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 348267us
I20240426 13:19:59.057482 25030 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.057739 24955 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:59.012588 26824 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.014906 27699 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240426 13:19:59.016726 25618 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Advancing to term 4
I20240426 13:19:59.067571 24949 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:59.067850 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.068639 24955 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:59.100771 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62236.2us
W20240426 13:19:59.143276 25070 leader_election.cc:334] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.686s (SENT)
I20240426 13:19:59.148610 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.023875 27835 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:19:59.152302 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.161847 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.032683 27867 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:59.042318 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.002890 27881 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.222s
W20240426 13:19:59.184332 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82600.9us
I20240426 13:19:59.159062 27640 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.219832 27778 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.003572 27883 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.263746 27883 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.264343 24951 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20240426 13:19:59.265112 24951 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:19:59.265329 24951 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240426 13:19:59.249814 27889 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 13:19:59.251621 27878 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20240426 13:19:59.275708 27888 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:59.282038 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1475) took 2665 ms. Trace:
I20240426 13:19:59.282550 25103 rpcz_store.cc:276] 0426 13:19:56.616768 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.616887 (+   119us) service_pool.cc:225] Handling call
0426 13:19:56.635070 (+ 18183us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.115212 (+480142us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:57.115247 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:57.115308 (+    61us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:57.115345 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:59.281747 (+2166402us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:59.281808 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:59.281959 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4285,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":1402,"spinlock_wait_cycles":1097023872}
I20240426 13:19:59.065174 27879 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.212s
I20240426 13:19:59.263901 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1596) took 3097 ms. Trace:
I20240426 13:19:59.265748 24951 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:59.226402 27835 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:59.268935 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.294606 24949 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.267367 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.291177 24954 rpcz_store.cc:276] 0426 13:19:56.166209 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.166605 (+   396us) service_pool.cc:225] Handling call
0426 13:19:59.263639 (+3097034us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:59.263808 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73691904}
I20240426 13:19:59.224314 27474 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.296777 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:59.224694 27894 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:19:59.297654 25024 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:59.309917 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.311159 25024 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:59.150765 27740 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:19:59.295962 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:59.300338 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.269742 27889 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.322657 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.285284 27350 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.327728 24949 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:59.334040 27894 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:59.206758 25621 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:19:59.365200 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 309792us
I20240426 13:19:59.164592 27376 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.128s
I20240426 13:19:59.296365 27835 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.292333 27724 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.096s
I20240426 13:19:59.415891 24949 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:59.392853 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 150286us
I20240426 13:19:59.393680 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.387475 27889 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.319974 27496 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.394762 27766 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.348480 27905 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20240426 13:19:59.391615 27796 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.428872 27796 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.429342 27796 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.373756 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.417124 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:19:59.395059 24954 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:59.432075 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58303.6us
I20240426 13:19:59.387516 27350 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:59.270629 26824 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.397171 27878 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20240426 13:19:59.434851 24955 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:19:59.402776 27835 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:59.394343 27892 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:59.444977 27911 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240426 13:19:59.431407 27796 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.447786 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.452605 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.454788 24948 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20240426 13:19:59.460114 25070 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 2.208s (SENT)
I20240426 13:19:59.460717 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.446660 27638 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:59.466157 27350 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.472755 27630 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.473192 27630 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:59.473387 27350 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.469653 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1674) took 1643 ms. Trace:
I20240426 13:19:59.474757 25101 rpcz_store.cc:276] 0426 13:19:57.826533 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.826686 (+   153us) service_pool.cc:225] Handling call
0426 13:19:57.827101 (+   415us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:58.299535 (+472434us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:58.299566 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:58.300121 (+   555us) log.cc:842] Serialized 52 byte log entry
0426 13:19:58.875281 (+575160us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:58.941123 (+ 65842us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.941570 (+   447us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.941788 (+   218us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.941885 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:59.469364 (+527479us) raft_consensus.cc:1671] finished
0426 13:19:59.469394 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:59.469580 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":624657,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"raft.queue_time_us":128457,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":272,"thread_start_us":761514,"threads_started":3,"wal-append.queue_time_us":682498}
I20240426 13:19:59.461727 27707 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:59.486799 25024 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.496387 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.496429 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.497495 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:19:59.464768 27881 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:59.465529 27638 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:59.501353 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:59.468163 27640 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:59.495306 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.048s	sys 0.084s
W20240426 13:19:59.502900 25100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.003s	sys 0.098s
W20240426 13:19:59.503122 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.048s	sys 0.084s
W20240426 13:19:59.503279 25100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.003s	sys 0.098s
W20240426 13:19:59.504985 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 574036us
I20240426 13:19:59.506275 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.507710 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.521677 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.522691 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.480794 27327 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20240426 13:19:59.504662 27638 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.504878 27707 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.528542 27482 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.528004 27766 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:59.474078 27630 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.581725 27916 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:59.580549 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.551283 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.460968 27892 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.614714 27707 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.618327 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95185.1us
W20240426 13:19:59.618046 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110428us
I20240426 13:19:59.623991 25024 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.584125 27881 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:59.629283 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.620939 27904 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20240426 13:19:59.640046 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.623558 27796 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:59.640425 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.620596 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.641235 27796 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:59.641374 27640 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:59.628896 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1486) took 5583 ms. Trace:
I20240426 13:19:59.641846 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.627918 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.002s	sys 0.023s
W20240426 13:19:59.646409 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.002s	sys 0.023s
W20240426 13:19:59.647722 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 607975us
I20240426 13:19:59.647526 25022 rpcz_store.cc:276] 0426 13:19:54.041039 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.800609 (+759570us) service_pool.cc:225] Handling call
0426 13:19:56.147917 (+1347308us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.171716 (+ 23799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:59.527112 (+3355396us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400055ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:59.624407 (+ 97295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961774976}
I20240426 13:19:59.649102 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.662249 27885 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20240426 13:19:59.662703 27908 consensus_queue.cc:1159] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
I20240426 13:19:59.691810 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.588047 27482 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.712162 27482 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:59.696370 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47340.9us
I20240426 13:19:59.637161 27916 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.644896 27796 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:59.651154 27676 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.000s
I20240426 13:19:59.647589 27640 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:59.704579 27855 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.826s	user 0.009s	sys 0.000s
W20240426 13:19:59.704707 27677 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.671s	user 0.004s	sys 0.004s
W20240426 13:19:59.707491 27850 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.671s	user 0.001s	sys 0.007s
W20240426 13:19:59.712363 27775 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.922s	user 0.011s	sys 0.000s
I20240426 13:19:59.722311 27850 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.723086 27775 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:19:59.723532 25618 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.559s	user 0.004s	sys 0.004s
W20240426 13:19:59.708144 27860 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s	user 0.000s	sys 0.005s
I20240426 13:19:59.715584 25096 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 5. Ops: []
I20240426 13:19:59.702028 25024 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:59.716503 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.719914 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:59.699154 27766 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:59.736048 25100 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:59.745234 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:59.745535 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.745801 25096 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.746555 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.737022 27850 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.733508 27677 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.721037 27676 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:59.737902 27916 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.739820 27835 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.740619 27714 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:59.742796 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.790009 27696 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.748220 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.825982 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83259.9us
W20240426 13:19:59.790748 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44261.1us
I20240426 13:19:59.826900 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.812618 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:59.792773 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.837894 25024 raft_consensus.cc:2362] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9. Re-sending same reply.
I20240426 13:19:59.805121 27482 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:59.834442 27492 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:59.834789 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2031) took 1224 ms. Trace:
I20240426 13:19:59.835150 27722 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:59.748948 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:19:59.745725 25618 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20240426 13:19:59.792905 27775 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.740650 27907 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:59.720824 27640 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:59.746212 27766 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.874884 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:59.890640 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 250259us
I20240426 13:19:59.839766 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.814769 27924 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20240426 13:19:59.826114 27714 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.872969 27492 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:59.926669 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86986.5us
W20240426 13:19:59.930773 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55985us
I20240426 13:19:59.930864 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.724232 27855 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.907647 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.891863 27799 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.040s	user 0.022s	sys 0.053s
I20240426 13:19:59.934818 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.927577 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.940448 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.927747 27766 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.941969 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.891570 25022 rpcz_store.cc:276] 0426 13:19:58.610439 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.691575 (+1081136us) service_pool.cc:225] Handling call
0426 13:19:59.834727 (+143152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:59.876277 27850 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:59.893600 25618 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 3
I20240426 13:19:59.911579 27907 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.935189 27492 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.990561 25024 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:59.815033 27740 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:59.834478 27696 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.942584 27799 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.087s	user 0.023s	sys 0.054s
I20240426 13:19:59.936602 27498 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.042409 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:00.013224 27867 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.759s	user 0.009s	sys 0.000s
I20240426 13:19:59.923249 27775 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.009753 27862 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.082s	user 0.005s	sys 0.004s
W20240426 13:20:00.056555 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46918.4us
I20240426 13:20:00.058787 25022 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.039458 25618 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240426 13:20:00.042927 27907 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:00.043485 27631 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:00.009979 27746 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.835983 27881 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:00.054167 27898 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:00.055159 27850 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.058734 27912 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.069509 27498 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:00.077270 27900 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.087384 27898 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.100615 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.061919 27740 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.065001 25618 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:00.103610 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:00.105736 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.062816 27867 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.117127 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 182377us
I20240426 13:20:00.119630 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.101331 27631 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:00.107684 27898 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:00.124106 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.037s	sys 0.087s
I20240426 13:20:00.099857 27892 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.123004 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.124897 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:00.126123 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.037s	sys 0.087s
I20240426 13:20:00.126700 25029 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.099537 27928 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:00.115703 27677 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.132102 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.067960 27862 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:00.121004 27900 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:00.135401 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.078792 27714 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.136304 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:00.142488 25022 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.125348 27835 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:00.146186 25030 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.133474 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:00.141009 27900 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.150388 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.135567 27631 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:00.141700 27677 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.155529 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.160282 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.149837 27714 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.164287 27886 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:00.164749 27886 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:00.165133 27886 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.150490 27855 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:00.150871 27932 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:20:00.158679 27889 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.727s	user 0.006s	sys 0.004s
I20240426 13:20:00.149037 27498 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.175730 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 777942us
W20240426 13:20:00.187057 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 245149us
I20240426 13:20:00.156849 27631 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:00.162159 27835 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:00.154481 27900 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:00.152472 27482 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.187817 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:00.176517 27925 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.371s	user 0.001s	sys 0.001s
W20240426 13:20:00.180716 27906 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20240426 13:20:00.180310 27926 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20240426 13:20:00.171579 27921 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20240426 13:20:00.204892 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:00.166543 27696 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:00.178557 27350 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.704s	user 0.009s	sys 0.001s
I20240426 13:20:00.172410 27855 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.207115 27482 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.194293 27631 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:00.213258 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.200680 27775 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.231267 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:00.242481 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80670.6us
I20240426 13:20:00.243330 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.243330 27722 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.163370 27740 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:00.245350 25022 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:00.246088 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2036) took 1568 ms. Trace:
I20240426 13:20:00.246549 25022 rpcz_store.cc:276] 0426 13:19:58.678106 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.124362 (+1446256us) service_pool.cc:225] Handling call
0426 13:20:00.246017 (+121655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:00.242729 27942 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.248587 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.249195 27943 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.250206 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.251173 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.189594 27889 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.195369 27835 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:00.251296 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63363.7us
I20240426 13:20:00.255214 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.220702 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:00.259737 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 510850us
W20240426 13:20:00.251224 27638 consensus_meta.cc:287] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.658s	user 0.011s	sys 0.000s
I20240426 13:20:00.243209 27940 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:00.154616 27850 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.210924 27936 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.269723 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.236428 27855 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.280774 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.281953 27935 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:00.250582 27903 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.202175 27932 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 13:20:00.283044 27903 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:00.234485 27696 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:00.271430 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.282652 27935 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.250825 27867 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.258086 27775 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:00.271667 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.275789 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.284924 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.287427 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:00.288252 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:00.288771 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.264909 27835 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:00.291005 25098 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.295789 25623 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:00.284372 24956 raft_consensus.cc:2362] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8. Re-sending same reply.
I20240426 13:20:00.293447 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.297101 24954 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:20:00.297315 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.299108 24949 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:00.301525 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:00.302253 24947 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.300453 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:00.310798 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62296.2us
I20240426 13:20:00.307145 24953 raft_consensus.cc:2362] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4. Re-sending same reply.
I20240426 13:20:00.291783 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:00.338850 27677 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.290343 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:00.342209 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.288161 27903 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:00.346710 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.294548 27775 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.293464 27862 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.357944 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.275775 27638 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.373816 27933 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:00.313385 27867 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:00.374297 27855 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.312414 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:00.315507 27722 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.340567 27677 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.379747 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:00.452819 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102677us
I20240426 13:20:00.454923 25098 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.157054 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:01.083119 25096 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.792s	user 0.009s	sys 0.000s
I20240426 13:20:01.158962 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:00.693982 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 402228us
W20240426 13:20:00.694377 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 404088us
I20240426 13:20:01.152907 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.291914 27916 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:00.354048 27903 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:01.237838 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:01.242529 25096 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:01.242846 24955 raft_consensus.cc:2362] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4. Re-sending same reply.
I20240426 13:20:01.243669 24949 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:01.243280 24948 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:01.151576 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:00.361780 27775 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:01.273295 24954 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:00.853013 27898 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.000s
I20240426 13:20:01.366011 27898 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.849782 27640 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.917s	user 0.004s	sys 0.005s
I20240426 13:20:01.370682 27640 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.848063 27766 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.896s	user 0.009s	sys 0.001s
I20240426 13:20:01.374495 27766 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.854799 27796 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.130s	user 0.008s	sys 0.001s
I20240426 13:20:01.279166 27638 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:01.379645 27796 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:20:01.391909 27714 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.223s	user 0.010s	sys 0.001s
W20240426 13:20:01.333931 24947 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.031s	user 0.007s	sys 0.003s
I20240426 13:20:01.426052 24947 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:01.426805 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1645) took 1139 ms (client timeout 1509 ms). Trace:
W20240426 13:20:01.427258 24947 rpcz_store.cc:269] 0426 13:20:00.286888 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.295484 (+  8596us) service_pool.cc:225] Handling call
0426 13:20:01.426732 (+1131248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:00.745041 27951 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20240426 13:20:01.465781 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.467687 24953 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:20:00.711488 27740 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:00.711555 27862 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.921543 27946 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:00.459204 27889 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:00.625193 27828 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 3.060s	user 0.003s	sys 0.003s
I20240426 13:20:01.293951 27933 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:00.624965 27791 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 3.983s	user 0.003s	sys 0.010s
I20240426 13:20:01.237825 27867 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:01.502357 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 260116us
W20240426 13:20:01.321554 27886 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.156s	user 0.005s	sys 0.006s
W20240426 13:20:01.315511 27492 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.256s	user 0.007s	sys 0.002s
W20240426 13:20:01.322026 27835 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.009s	user 0.010s	sys 0.000s
W20240426 13:20:01.315447 27907 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.233s	user 0.006s	sys 0.004s
W20240426 13:20:01.324808 27631 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.088s	user 0.009s	sys 0.000s
I20240426 13:20:01.526332 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.562187 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:01.561996 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.569262 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.658295 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1961) took 1376 ms. Trace:
I20240426 13:20:01.659933 24947 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.589684 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1960) took 1307 ms. Trace:
I20240426 13:20:01.669533 24953 rpcz_store.cc:276] 0426 13:20:00.281838 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.245014 (+963176us) service_pool.cc:225] Handling call
0426 13:20:01.589600 (+344586us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.415977 27916 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:01.673519 24948 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:20:01.660593 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1643) took 1374 ms. Trace:
I20240426 13:20:01.684736 24947 rpcz_store.cc:276] 0426 13:20:00.286232 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.525991 (+1239759us) service_pool.cc:225] Handling call
0426 13:20:01.660523 (+134532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.658843 24949 rpcz_store.cc:276] 0426 13:20:00.282060 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.461564 (+1179504us) service_pool.cc:225] Handling call
0426 13:20:01.658220 (+196656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.660975 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1644) took 1374 ms. Trace:
I20240426 13:20:01.687211 24955 rpcz_store.cc:276] 0426 13:20:00.286574 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.400373 (+1113799us) service_pool.cc:225] Handling call
0426 13:20:01.660907 (+260534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.690492 24954 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.690941 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1646) took 1401 ms. Trace:
I20240426 13:20:01.691187 24954 rpcz_store.cc:276] 0426 13:20:00.289610 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.347948 (+ 58338us) service_pool.cc:225] Handling call
0426 13:20:01.690888 (+1342940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.693250 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 26) took 7223 ms. Trace:
I20240426 13:20:01.693769 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1962) took 1411 ms. Trace:
I20240426 13:20:01.964326 24948 rpcz_store.cc:276] 0426 13:20:00.282461 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.400689 (+1118228us) service_pool.cc:225] Handling call
0426 13:20:01.693705 (+293016us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:01.966089 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.010s	sys 0.018s
W20240426 13:20:01.966295 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 728522us
W20240426 13:20:01.641439 27900 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.430s	user 0.015s	sys 0.027s
W20240426 13:20:01.843132 25022 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.000s	sys 0.032s
I20240426 13:20:01.967828 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:01.976801 24949 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:01.977699 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1965) took 1623 ms. Trace:
I20240426 13:20:01.977905 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:01.978133 24949 rpcz_store.cc:276] 0426 13:20:00.354660 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.694746 (+1340086us) service_pool.cc:225] Handling call
0426 13:20:01.969364 (+274618us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:01.977409 (+  8045us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:01.977641 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:01.862890 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.008s	sys 0.002s
W20240426 13:20:01.979796 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s	user 0.008s	sys 0.006s
W20240426 13:20:01.966614 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.010s	sys 0.018s
W20240426 13:20:01.943967 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.015s	sys 0.019s
I20240426 13:20:01.693758 25097 rpcz_store.cc:276] 0426 13:19:54.458202 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.459851 (+  1649us) service_pool.cc:225] Handling call
0426 13:20:01.681354 (+7221503us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:01.681474 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5705892096}
W20240426 13:20:01.980818 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.016s	sys 0.019s
W20240426 13:20:01.963579 27944 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.318s	user 0.000s	sys 0.001s
W20240426 13:20:01.984210 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.000s	sys 0.006s
W20240426 13:20:01.984694 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.000s	sys 0.006s
W20240426 13:20:01.953045 27954 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20240426 13:20:01.624814 27498 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.873s	user 0.001s	sys 0.011s
I20240426 13:20:01.576426 27862 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:01.976866 25022 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.001s	sys 0.034s
I20240426 13:20:01.807202 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 532) took 1414 ms. Trace:
I20240426 13:20:01.997927 24955 rpcz_store.cc:276] 0426 13:20:00.392622 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.687761 (+1295139us) service_pool.cc:225] Handling call
0426 13:20:01.806536 (+118775us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:01.806775 (+   239us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:01.806812 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:01.806868 (+    56us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:01.806917 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:01.806982 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:01.807124 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.561935 27939 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.567806 27740 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:01.992475 27957 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:01.671594 27913 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:01.888820 27947 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20240426 13:20:01.578720 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 579) took 1285 ms. Trace:
I20240426 13:20:01.574151 27933 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.006511 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 378) took 4819 ms. Trace:
I20240426 13:20:01.692927 26824 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:01.746657 27303 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.591395 27945 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:02.020514 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.674s (SENT) [suppressed 29 similar messages]
I20240426 13:20:01.685978 27492 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:02.002322 27498 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.263s	user 0.003s	sys 0.014s
W20240426 13:20:02.047060 27948 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20240426 13:20:01.987761 27898 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.030454 27867 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.025503 27960 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.837852 27950 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:02.030239 25617 rpcz_store.cc:276] 0426 13:20:00.281090 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.284576 (+  3486us) service_pool.cc:225] Handling call
0426 13:20:01.566095 (+1281519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.676788 27631 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.671504 27886 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.009843 27850 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.039353 27686 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:02.039409 27303 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.687515 27835 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.931010 27935 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:02.044764 27820 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 4.979s	user 0.025s	sys 0.043s
I20240426 13:20:01.990379 27900 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.017954 27916 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.012925 27796 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.030370 27913 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.011003 27862 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.843211 27962 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.044209 27640 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.059007 27740 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.676508 27907 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.062017 27686 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.072364 27303 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:01.718205 27903 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:02.049316 27945 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.056864 27898 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.016144 27939 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.058028 27950 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.034955 25022 rpcz_store.cc:276] 0426 13:19:57.187602 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.276316 (+3088714us) service_pool.cc:225] Handling call
0426 13:20:00.296402 (+ 20086us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:00.309528 (+ 13126us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:00.309573 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:00.320909 (+ 11336us) log.cc:842] Serialized 26 byte log entry
0426 13:20:01.154020 (+833111us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:01.997538 (+843518us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.006038 (+  8500us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.006146 (+   108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.006209 (+    63us) raft_consensus.cc:1671] finished
0426 13:20:02.006228 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.006418 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20716,"raft.queue_time_us":392942,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":250530,"thread_start_us":718705,"threads_started":2,"wal-append.queue_time_us":1365146}
I20240426 13:20:02.073969 27835 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.324378 27710 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.275538 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1489) took 8198 ms (client timeout 1661 ms). Trace:
W20240426 13:20:02.216799 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1482) took 8202 ms (client timeout 1527 ms). Trace:
W20240426 13:20:02.274582 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2029) took 3664 ms (client timeout 2065 ms). Trace:
I20240426 13:20:02.269470 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2012) took 6775 ms. Trace:
I20240426 13:20:02.227897 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 536) took 1074 ms. Trace:
W20240426 13:20:02.271189 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1511) took 4261 ms (client timeout 2040 ms). Trace:
I20240426 13:20:02.226070 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 534) took 1779 ms. Trace:
W20240426 13:20:02.273290 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2028) took 3663 ms (client timeout 2832 ms). Trace:
I20240426 13:20:02.206857 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1593) took 6535 ms. Trace:
I20240426 13:20:02.344895 24948 rpcz_store.cc:276] 0426 13:20:00.446932 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.980437 (+1533505us) service_pool.cc:225] Handling call
0426 13:20:02.224890 (+244453us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.225454 (+   564us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.225509 (+    55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.225573 (+    64us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:02.225629 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.225719 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.225879 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.342797 25023 rpcz_store.cc:276] 0426 13:19:55.493590 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.736541 (+1242951us) service_pool.cc:225] Handling call
0426 13:19:56.736894 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.174225 (+437331us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.268824 (+5094599us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.269256 (+   432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993131776}
I20240426 13:20:02.326491 27916 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.342963 24955 rpcz_store.cc:276] 0426 13:20:01.152878 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.999398 (+846520us) service_pool.cc:225] Handling call
0426 13:20:02.226885 (+227487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.227326 (+   441us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.227372 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.227448 (+    76us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:02.227500 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.227571 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.227701 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:02.345069 25024 rpcz_store.cc:269] 0426 13:19:58.609633 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.839743 (+1230110us) service_pool.cc:225] Handling call
0426 13:20:02.272762 (+2433019us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.273059 (+   297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122441472}
W20240426 13:20:02.342295 25031 rpcz_store.cc:269] 0426 13:19:54.013909 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.026563 (+ 12654us) service_pool.cc:225] Handling call
0426 13:20:02.215778 (+8189215us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.216504 (+   726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4754335616}
W20240426 13:20:02.343194 25027 rpcz_store.cc:269] 0426 13:19:58.009803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.025261 (+ 15458us) service_pool.cc:225] Handling call
0426 13:20:02.270769 (+4245508us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.271019 (+   250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":277254400}
W20240426 13:20:02.342446 25029 rpcz_store.cc:269] 0426 13:19:58.609922 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.161298 (+551376us) service_pool.cc:225] Handling call
0426 13:20:02.274104 (+3112806us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.274377 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2215829888}
I20240426 13:20:02.288398 27708 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.342137 25025 rpcz_store.cc:269] 0426 13:19:54.077519 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.208860 (+131341us) service_pool.cc:225] Handling call
0426 13:20:02.275101 (+8066241us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.275353 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5057163648}
W20240426 13:20:02.361647 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 390) took 2073 ms (client timeout 1653 ms). Trace:
I20240426 13:20:02.371979 25027 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:02.067894 27886 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.372731 25023 rpcz_store.cc:269] 0426 13:20:00.287882 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.361245 (+2073363us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:02.361542 (+   297us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:02.068464 27631 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.379739 25028 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:02.371487 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.353302 27710 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.385082 27907 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.385165 25029 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:02.077198 27492 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.322953 27933 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.390494 25101 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:02.266455 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:02.365312 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.010s	sys 0.007s
I20240426 13:20:02.399571 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1495) took 4589 ms. Trace:
I20240426 13:20:02.400151 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1497) took 4589 ms. Trace:
I20240426 13:20:02.403370 25029 rpcz_store.cc:276] 0426 13:19:57.801943 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.384241 (+4582298us) service_pool.cc:225] Handling call
0426 13:20:02.384866 (+   625us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.386462 (+  1596us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.391639 (+  5177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.402614 25027 rpcz_store.cc:276] 0426 13:19:57.801391 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.370790 (+4569399us) service_pool.cc:225] Handling call
0426 13:20:02.371610 (+   820us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.388811 (+ 17201us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.391132 (+  2321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71296}
I20240426 13:20:02.399035 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:02.403064 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.011s	sys 0.007s
I20240426 13:20:02.063642 27960 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.407878 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:02.402969 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.000s	sys 0.060s
I20240426 13:20:02.391214 27710 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.412199 25097 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:02.413674 27933 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.416682 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.000s	sys 0.061s
W20240426 13:20:02.417716 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.011s	sys 0.042s
I20240426 13:20:02.421113 24949 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:02.422346 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.011s	sys 0.043s
I20240426 13:20:02.421686 25096 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:02.425397 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:02.426993 24953 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:02.427189 27903 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:02.428416 26996 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:02.429028 26996 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:02.429226 27913 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.427191 27945 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.430755 26996 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.431571 27683 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.432569 27683 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.435390 27683 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.442713 27916 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.445574 27916 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.448004 27900 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.448429 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.448944 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:02.423266 27946 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.010s	sys 0.026s
I20240426 13:20:02.346225 24950 rpcz_store.cc:276] 0426 13:19:55.670699 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.670817 (+   118us) service_pool.cc:225] Handling call
0426 13:19:55.671080 (+   263us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.728336 (+ 57256us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.728382 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.775242 (+ 46860us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.400043 (+624801us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:56.400081 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.400148 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:00.294445 (+3894297us) spinlock_profiling.cc:243] Waited 8.84 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:02.205870 (+1911425us) raft_consensus.cc:1671] finished
0426 13:20:02.205949 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.206487 (+   538us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56865,"raft.queue_time_us":63068,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":121419,"spinlock_wait_cycles":20340608,"thread_start_us":243076,"threads_started":1,"wal-append.queue_time_us":446915}
I20240426 13:20:02.464831 25617 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:02.469115 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.461627 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1963) took 2169 ms. Trace:
W20240426 13:20:02.471001 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45497.3us
I20240426 13:20:02.471850 24953 rpcz_store.cc:276] 0426 13:20:00.282749 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.688227 (+1405478us) service_pool.cc:225] Handling call
0426 13:20:02.265947 (+577720us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:02.451705 (+185758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3938176}
I20240426 13:20:02.472831 27913 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240426 13:20:02.477326 27913 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:02.477783 27913 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.464926 27960 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.457376 27946 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.483397 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:02.479638 27472 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.004s
I20240426 13:20:02.492206 27736 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:02.492748 27736 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:02.481904 27976 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20240426 13:20:02.503927 26996 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:02.504688 24950 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:02.453639 27708 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.505558 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1968) took 1344 ms. Trace:
I20240426 13:20:02.505981 24950 rpcz_store.cc:276] 0426 13:20:01.161129 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.473422 (+1312293us) service_pool.cc:225] Handling call
0426 13:20:02.500354 (+ 26932us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.505183 (+  4829us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.505463 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.508052 24950 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:02.508292 27886 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.508734 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1969) took 1347 ms. Trace:
I20240426 13:20:02.508868 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 508) took 3721 ms. Trace:
I20240426 13:20:02.509178 27886 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.509930 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.510277 24953 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:02.510573 27710 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.504932 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:02.479342 27959 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:20:02.511471 27710 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.512653 27886 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.511013 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1970) took 1349 ms. Trace:
I20240426 13:20:02.509370 25102 rpcz_store.cc:276] 0426 13:19:58.787589 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.824165 (+ 36576us) service_pool.cc:225] Handling call
0426 13:19:58.824538 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.868205 (+ 43667us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:58.868236 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:58.868279 (+    43us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:58.868309 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.508350 (+3640041us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b54003ee9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.508411 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.508680 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99872000}
I20240426 13:20:02.513511 24953 rpcz_store.cc:276] 0426 13:20:01.161698 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.509522 (+1347824us) service_pool.cc:225] Handling call
0426 13:20:02.510008 (+   486us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.510748 (+   740us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.510940 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.515794 27970 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.509207 24950 rpcz_store.cc:276] 0426 13:20:01.161405 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.507154 (+1345749us) service_pool.cc:225] Handling call
0426 13:20:02.507723 (+   569us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.508442 (+   719us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.508652 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.516518 27970 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.509284 27492 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.522089 27492 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.523286 27640 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:02.525188 27492 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.525310 27710 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:02.525826 27775 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.108s	user 0.015s	sys 0.004s
I20240426 13:20:02.526420 27775 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.527350 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 574) took 3570 ms. Trace:
I20240426 13:20:02.527941 25618 rpcz_store.cc:276] 0426 13:19:58.956727 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.956850 (+   123us) service_pool.cc:225] Handling call
0426 13:19:58.957178 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:00.120155 (+1162977us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.526721 (+2406566us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.527122 (+   401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111197568}
I20240426 13:20:02.535566 27970 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.536757 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.535468 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.530078 24947 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:02.539793 27844 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:20:02.539942 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 417) took 2254 ms (client timeout 1683 ms). Trace:
I20240426 13:20:02.502094 27898 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.543140 24948 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:02.547377 25624 rpcz_store.cc:269] 0426 13:20:00.285542 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.285672 (+   130us) service_pool.cc:225] Handling call
0426 13:20:02.535071 (+2249399us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:02.539844 (+  4773us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6336256}
W20240426 13:20:02.550755 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49919.3us
I20240426 13:20:02.541399 27913 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.568609 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1966) took 1407 ms. Trace:
I20240426 13:20:02.556208 27862 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.565701 27898 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.569953 24948 rpcz_store.cc:276] 0426 13:20:01.160513 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.424668 (+1264155us) service_pool.cc:225] Handling call
0426 13:20:02.478901 (+ 54233us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.566162 (+ 87261us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.568218 (+  2056us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.567886 27886 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20240426 13:20:02.585346 27775 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.585086 27862 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.597824 27941 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.236s	user 0.003s	sys 0.031s
I20240426 13:20:02.594482 27913 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:02.600209 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 90470.8us
I20240426 13:20:02.602699 27487 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:02.588189 27950 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.609059 24955 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:02.609666 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1967) took 1448 ms. Trace:
I20240426 13:20:02.588495 27482 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.611192 24955 rpcz_store.cc:276] 0426 13:20:01.160840 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.608165 (+1447325us) service_pool.cc:225] Handling call
0426 13:20:02.608616 (+   451us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.609420 (+   804us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.609619 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.614970 25022 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:02.615716 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2025) took 4007 ms. Trace:
I20240426 13:20:02.620213 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 535) took 1468 ms. Trace:
I20240426 13:20:02.620570 24949 rpcz_store.cc:276] 0426 13:20:01.152188 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.993031 (+840843us) service_pool.cc:225] Handling call
0426 13:20:01.993455 (+   424us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.440861 (+447406us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.619861 (+179000us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.620145 (+   284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":969883008}
I20240426 13:20:02.615978 25022 rpcz_store.cc:276] 0426 13:19:58.608787 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.614095 (+4005308us) service_pool.cc:225] Handling call
0426 13:20:02.614569 (+   474us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.615456 (+   887us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.615658 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.622198 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.623118 25022 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:02.624428 27656 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:02.626143 27513 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:02.626618 27513 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.627786 27513 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:02.628886 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 384) took 3854 ms (client timeout 1924 ms). Trace:
I20240426 13:20:02.628939 27513 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:02.629530 25030 rpcz_store.cc:269] 0426 13:19:58.774710 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.299817 (+525107us) service_pool.cc:225] Handling call
0426 13:20:02.628486 (+3328669us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b5400242f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.628699 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1907150208}
I20240426 13:20:02.630671 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.632548 27715 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:02.633083 27715 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:02.634603 27975 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.205s	user 0.001s	sys 0.001s
I20240426 13:20:02.623975 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.637128 25022 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:02.640935 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139280us
I20240426 13:20:02.643112 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.642062 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.644762 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.608729 27946 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:02.649122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 46841.1us
I20240426 13:20:02.649755 27855 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.649998 27722 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.617580 27939 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.655484 27862 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.656430 27686 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.656955 27482 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.684258 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.685575 27482 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.677102 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:02.686093 27722 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.687201 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:02.688325 27656 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:02.688655 27722 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.688872 27656 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.689517 27656 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:02.677500 25022 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:02.692567 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:02.698688 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.699420 27687 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.703336 27724 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:02.702661 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.704016 27950 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:02.704720 27950 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:02.704792 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:02.705143 27950 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.705950 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.709758 27868 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:02.709951 25096 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:02.710287 27868 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:02.708640 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.710428 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.726805 25026 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:02.731690 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:02.735834 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93576.6us
I20240426 13:20:02.716411 27715 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.743781 27715 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.748512 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 522) took 2849 ms. Trace:
I20240426 13:20:02.735224 27487 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:02.749222 27487 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:02.749814 27487 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:02.748927 24952 rpcz_store.cc:276] 0426 13:19:59.899096 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.899237 (+   141us) service_pool.cc:225] Handling call
0426 13:19:59.899631 (+   394us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:00.218798 (+319167us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:00.218826 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:00.221326 (+  2500us) log.cc:842] Serialized 26 byte log entry
0426 13:20:00.311801 (+ 90475us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:00.311841 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.747925 (+2436084us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.747988 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.748058 (+    70us) raft_consensus.cc:1671] finished
0426 13:20:02.748072 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.748330 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14876,"raft.queue_time_us":86888,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":20534,"spinlock_wait_cycles":404918016,"thread_start_us":232846,"threads_started":2,"wal-append.queue_time_us":1354832}
I20240426 13:20:02.748179 27656 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.753832 27862 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.753772 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.756156 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 533) took 2328 ms. Trace:
I20240426 13:20:02.757611 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.753649 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.757688 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.762636 25097 raft_consensus.cc:595] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.776189 25097 raft_consensus.cc:603] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:02.764672 27833 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:02.755807 27950 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.779732 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.775317 24954 rpcz_store.cc:276] 0426 13:20:00.427321 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.691632 (+1264311us) service_pool.cc:225] Handling call
0426 13:20:02.755027 (+1063395us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:02.755348 (+   321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.755691 (+   343us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.755719 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.755758 (+    39us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:02.755786 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.755839 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.755959 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2396544}
I20240426 13:20:02.780547 25030 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.711977 27855 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.782027 25022 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:02.792312 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1488) took 5657 ms (client timeout 1886 ms). Trace:
I20240426 13:20:02.792574 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:02.792934 25105 rpcz_store.cc:269] 0426 13:19:57.134210 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.135173 (+   963us) service_pool.cc:225] Handling call
0426 13:20:02.791718 (+5656545us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.792009 (+   291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1536800640}
I20240426 13:20:02.794314 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:02.795020 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1971) took 1633 ms. Trace:
I20240426 13:20:02.795439 24953 rpcz_store.cc:276] 0426 13:20:01.161963 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.528133 (+1366170us) service_pool.cc:225] Handling call
0426 13:20:02.559544 (+ 31411us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.560159 (+   615us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.560196 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:02.560944 (+   748us) log.cc:842] Serialized 26 byte log entry
0426 13:20:02.697832 (+136888us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:02.697876 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.697939 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.794741 (+ 96802us) raft_consensus.cc:1671] finished
0426 13:20:02.794774 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.794940 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":136436,"threads_started":1,"wal-append.queue_time_us":152082}
I20240426 13:20:02.796790 27766 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.795061 25096 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:02.783205 27833 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.799134 27487 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.800704 25624 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:02.787876 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.822325 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:02.822563 27487 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.823499 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.823727 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.829244 27487 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:02.785264 27768 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.836884 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49813.1us
I20240426 13:20:02.848563 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1964) took 2565 ms. Trace:
I20240426 13:20:02.849107 24947 rpcz_store.cc:276] 0426 13:20:00.283036 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.694986 (+1411950us) service_pool.cc:225] Handling call
0426 13:20:01.731220 (+ 36234us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.553235 (+822015us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.832268 (+279033us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.848485 (+ 16217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1832775936}
I20240426 13:20:02.861980 25096 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:03.393020 27923 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:03.186132 27868 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:03.393991 27923 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:04.096516 27980 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.575s	user 0.001s	sys 0.000s
I20240426 13:20:04.080456 27855 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:04.146396 27981 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.358s	user 0.002s	sys 0.000s
W20240426 13:20:04.418555 27981 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20240426 13:20:04.458138 27868 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.473433 27950 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:04.484431 27984 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.664s	user 0.000s	sys 0.002s
I20240426 13:20:04.525707 25031 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.526314 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1499) took 6720 ms. Trace:
I20240426 13:20:04.526540 25031 rpcz_store.cc:276] 0426 13:19:57.806206 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.525197 (+6718991us) service_pool.cc:225] Handling call
0426 13:20:04.525554 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.526107 (+   553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.526254 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:04.527370 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2073) took 1675 ms (client timeout 1660 ms). Trace:
W20240426 13:20:04.527644 25031 rpcz_store.cc:269] 0426 13:20:02.851985 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.527227 (+1675242us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:04.527317 (+    90us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:04.528510 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.529534 25031 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20240426 13:20:04.533878 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 395) took 2020 ms (client timeout 2031 ms). Trace:
W20240426 13:20:04.534142 25031 rpcz_store.cc:269] 0426 13:20:02.513101 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.528238 (+2015137us) service_pool.cc:225] Handling call
0426 13:20:04.533806 (+  5568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.535084 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:04.535876 25031 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.540807 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:04.541168 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2056) took 2010 ms (client timeout 2090 ms). Trace:
W20240426 13:20:04.542984 25031 rpcz_store.cc:269] 0426 13:20:02.530971 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.534844 (+2003873us) service_pool.cc:225] Handling call
0426 13:20:04.541117 (+  6273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.544806 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:04.545580 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2068) took 1897 ms (client timeout 1993 ms). Trace:
W20240426 13:20:04.546561 25031 rpcz_store.cc:269] 0426 13:20:02.648664 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.544534 (+1895870us) service_pool.cc:225] Handling call
0426 13:20:04.545530 (+   996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.547470 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.548082 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:04.548545 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 396) took 1927 ms (client timeout 2035 ms). Trace:
W20240426 13:20:04.548885 25031 rpcz_store.cc:269] 0426 13:20:02.621325 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.547181 (+1925856us) service_pool.cc:225] Handling call
0426 13:20:04.548498 (+  1317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.549814 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:04.550546 25031 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:04.559969 25031 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:20:04.561026 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2058) took 2029 ms (client timeout 2193 ms). Trace:
W20240426 13:20:04.561465 25031 rpcz_store.cc:269] 0426 13:20:02.531624 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.549542 (+2017918us) service_pool.cc:225] Handling call
0426 13:20:04.560967 (+ 11425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.563057 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.563809 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2070) took 1836 ms. Trace:
I20240426 13:20:04.564201 25031 rpcz_store.cc:276] 0426 13:20:02.727478 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.562738 (+1835260us) service_pool.cc:225] Handling call
0426 13:20:04.563760 (+  1022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.565521 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.566161 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2035) took 5892 ms. Trace:
I20240426 13:20:04.566542 25031 rpcz_store.cc:276] 0426 13:19:58.673511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.564827 (+5891316us) service_pool.cc:225] Handling call
0426 13:20:04.565232 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.565830 (+   598us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.566111 (+   281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.568146 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 380) took 5891 ms. Trace:
I20240426 13:20:04.568465 25031 rpcz_store.cc:276] 0426 13:19:58.676798 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.567217 (+5890419us) service_pool.cc:225] Handling call
0426 13:20:04.567588 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.567805 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.567836 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.567881 (+    45us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:04.567911 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.567961 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.568082 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.570022 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20240426 13:20:04.570700 25031 pending_rounds.cc:76] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:20:04.570952 25031 pending_rounds.cc:97] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240426 13:20:04.571192 25031 raft_consensus.cc:2866] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.571817 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 381) took 5894 ms. Trace:
I20240426 13:20:04.572217 25031 rpcz_store.cc:276] 0426 13:19:58.677095 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.569279 (+5892184us) service_pool.cc:225] Handling call
0426 13:20:04.569621 (+   342us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.571611 (+  1990us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.571762 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.582791 27928 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:04.635057 27962 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.635814 27962 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.638080 27962 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:04.639407 27716 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.640004 27716 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.642200 27716 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.638609 27962 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.644140 27962 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.646884 27962 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:04.663463 25030 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.882s	user 0.011s	sys 0.018s
I20240426 13:20:04.664605 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:04.665345 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2059) took 2133 ms (client timeout 1941 ms). Trace:
I20240426 13:20:04.666373 25031 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:04.666299 25030 rpcz_store.cc:269] 0426 13:20:02.531874 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.779465 (+247591us) service_pool.cc:225] Handling call
0426 13:20:04.665248 (+1885783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.668303 27656 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.668519 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 385) took 5893 ms. Trace:
I20240426 13:20:04.668846 25030 rpcz_store.cc:276] 0426 13:19:58.776002 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.667599 (+5891597us) service_pool.cc:225] Handling call
0426 13:20:04.667942 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.668196 (+   254us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.668219 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.668272 (+    53us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:04.668302 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.668354 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.668469 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.670658 25030 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 10. Ops: []
I20240426 13:20:04.671293 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 386) took 5742 ms. Trace:
I20240426 13:20:04.671631 25030 rpcz_store.cc:276] 0426 13:19:58.928987 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.670164 (+5741177us) service_pool.cc:225] Handling call
0426 13:20:04.670485 (+   321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.671070 (+   585us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.671244 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.698518 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2037) took 5304 ms. Trace:
I20240426 13:20:04.699075 25030 rpcz_store.cc:276] 0426 13:19:59.394257 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.673090 (+5278833us) service_pool.cc:225] Handling call
0426 13:20:04.673432 (+   342us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.674103 (+   671us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.674132 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.674609 (+   477us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.697974 (+ 23365us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:04.698020 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.698292 (+   272us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.698340 (+    48us) raft_consensus.cc:1671] finished
0426 13:20:04.698355 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.698463 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":22975,"threads_started":1,"wal-append.queue_time_us":20469}
I20240426 13:20:04.701490 25030 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:04.703655 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2038) took 5309 ms. Trace:
I20240426 13:20:04.704107 25030 rpcz_store.cc:276] 0426 13:19:59.394551 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.700752 (+5306201us) service_pool.cc:225] Handling call
0426 13:20:04.701196 (+   444us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.701922 (+   726us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.703588 (+  1666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.705611 25030 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:04.706293 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2039) took 5311 ms. Trace:
I20240426 13:20:04.706648 25030 rpcz_store.cc:276] 0426 13:19:59.394841 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.704980 (+5310139us) service_pool.cc:225] Handling call
0426 13:20:04.705381 (+   401us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.706059 (+   678us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.706241 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.710042 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1507) took 6729 ms. Trace:
I20240426 13:20:04.710507 25025 rpcz_store.cc:276] 0426 13:19:57.980553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.708853 (+6728300us) service_pool.cc:225] Handling call
0426 13:20:04.709181 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.709608 (+   427us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.709650 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.709708 (+    58us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.709753 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.709845 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.709980 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.712340 25025 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:04.713009 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2041) took 5317 ms. Trace:
I20240426 13:20:04.713372 25025 rpcz_store.cc:276] 0426 13:19:59.395419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.711681 (+5316262us) service_pool.cc:225] Handling call
0426 13:20:04.712109 (+   428us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.712752 (+   643us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.712950 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.715032 25025 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.717656 25029 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.718492 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1509) took 6719 ms. Trace:
I20240426 13:20:04.718881 25029 rpcz_store.cc:276] 0426 13:19:57.999507 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.716968 (+6717461us) service_pool.cc:225] Handling call
0426 13:20:04.717423 (+   455us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.718210 (+   787us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.718426 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:04.724769 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 519) took 2338 ms (client timeout 1653 ms). Trace:
W20240426 13:20:04.725382 25099 rpcz_store.cc:269] 0426 13:20:02.385870 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.406102 (+ 20232us) service_pool.cc:225] Handling call
0426 13:20:04.724419 (+2318317us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.724649 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":957015936}
I20240426 13:20:04.715776 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2042) took 5320 ms. Trace:
I20240426 13:20:04.729290 25025 rpcz_store.cc:276] 0426 13:19:59.395680 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.714198 (+5318518us) service_pool.cc:225] Handling call
0426 13:20:04.714669 (+   471us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.715484 (+   815us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.715719 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.730902 25025 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:04.731763 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2045) took 5335 ms. Trace:
I20240426 13:20:04.732270 25025 rpcz_store.cc:276] 0426 13:19:59.396522 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.730191 (+5333669us) service_pool.cc:225] Handling call
0426 13:20:04.730625 (+   434us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.731424 (+   799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.731622 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.733728 25025 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:04.734378 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 387) took 5259 ms. Trace:
I20240426 13:20:04.734766 25025 rpcz_store.cc:276] 0426 13:19:59.475426 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.733029 (+5257603us) service_pool.cc:225] Handling call
0426 13:20:04.733362 (+   333us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.734099 (+   737us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.734317 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.738579 25025 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:04.738720 27911 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.739192 27911 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.739303 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2050) took 5253 ms. Trace:
I20240426 13:20:04.739708 25025 rpcz_store.cc:276] 0426 13:19:59.486198 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.737954 (+5251756us) service_pool.cc:225] Handling call
0426 13:20:04.738320 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.739075 (+   755us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.739257 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.740473 27911 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.741469 25025 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.741983 27911 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.742184 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2051) took 5255 ms. Trace:
I20240426 13:20:04.742923 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:04.742682 25025 rpcz_store.cc:276] 0426 13:19:59.486463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.740664 (+5254201us) service_pool.cc:225] Handling call
0426 13:20:04.741034 (+   370us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.741946 (+   912us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.742118 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.743592 25623 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:04.756505 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2040) took 5361 ms. Trace:
I20240426 13:20:04.757104 25030 rpcz_store.cc:276] 0426 13:19:59.395144 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.707537 (+5312393us) service_pool.cc:225] Handling call
0426 13:20:04.707917 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.708618 (+   701us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.708645 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.709121 (+   476us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.755763 (+ 46642us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:04.755834 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.755910 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.755966 (+    56us) raft_consensus.cc:1671] finished
0426 13:20:04.755978 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.756422 (+   444us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":46201,"threads_started":1,"wal-append.queue_time_us":35451}
I20240426 13:20:04.759290 25030 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:04.760447 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2053) took 5122 ms. Trace:
I20240426 13:20:04.760864 25030 rpcz_store.cc:276] 0426 13:19:59.638527 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.758691 (+5120164us) service_pool.cc:225] Handling call
0426 13:20:04.759104 (+   413us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.760224 (+  1120us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.760392 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.763392 25623 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:04.764124 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 427) took 1925 ms. Trace:
I20240426 13:20:04.764447 25623 rpcz_store.cc:276] 0426 13:20:02.838558 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.844428 (+  5870us) service_pool.cc:225] Handling call
0426 13:20:04.742648 (+1898220us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:04.764011 (+ 21363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14691200}
I20240426 13:20:04.766175 27708 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.783094 27725 raft_consensus.cc:465] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Not starting pre-election -- already a leader
I20240426 13:20:04.783679 27725 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.784194 27725 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.787004 27725 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.793334 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 389) took 5288 ms. Trace:
W20240426 13:20:04.794826 24995 connection.cc:232] RPC call timeout handler was delayed by -1.8494979160000184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2959s for a response
I20240426 13:20:04.793946 25025 rpcz_store.cc:276] 0426 13:19:59.505358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.744152 (+5238794us) service_pool.cc:225] Handling call
0426 13:20:04.746772 (+  2620us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.747552 (+   780us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.747583 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.748076 (+   493us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.770793 (+ 22717us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.792167 (+ 21374us) log.cc:842] Serialized 14 byte log entry
0426 13:20:04.792953 (+   786us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.793078 (+   125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.793127 (+    49us) raft_consensus.cc:1671] finished
0426 13:20:04.793141 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.793276 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":517,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":43266,"threads_started":2,"wal-append.queue_time_us":22187}
W20240426 13:20:04.796396 24995 connection.cc:232] RPC call timeout handler was delayed by -1.3009583929999735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20240426 13:20:04.798056 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 392) took 2585 ms. Trace:
I20240426 13:20:04.798434 25025 rpcz_store.cc:276] 0426 13:20:02.212939 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.797157 (+2584218us) service_pool.cc:225] Handling call
0426 13:20:04.797507 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.797735 (+   228us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.797763 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.797808 (+    45us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:04.797845 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.797900 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.798005 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.803896 25025 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:04.804553 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 394) took 2291 ms. Trace:
I20240426 13:20:04.804904 25025 rpcz_store.cc:276] 0426 13:20:02.512745 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.803184 (+2290439us) service_pool.cc:225] Handling call
0426 13:20:04.803522 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.804311 (+   789us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.804487 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.806794 25025 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:20:04.807415 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2057) took 2276 ms. Trace:
I20240426 13:20:04.807788 25025 rpcz_store.cc:276] 0426 13:20:02.531333 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.805982 (+2274649us) service_pool.cc:225] Handling call
0426 13:20:04.806371 (+   389us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.807164 (+   793us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.807361 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.809327 25025 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:04.809896 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2069) took 2105 ms. Trace:
I20240426 13:20:04.810248 25025 rpcz_store.cc:276] 0426 13:20:02.704433 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.808599 (+2104166us) service_pool.cc:225] Handling call
0426 13:20:04.808929 (+   330us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.809676 (+   747us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.809845 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.811740 25025 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.812402 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 397) took 2066 ms. Trace:
I20240426 13:20:04.812726 25025 rpcz_store.cc:276] 0426 13:20:02.745852 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.810993 (+2065141us) service_pool.cc:225] Handling call
0426 13:20:04.811469 (+   476us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.812206 (+   737us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.812356 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.814450 25025 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:04.815080 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2071) took 2026 ms. Trace:
I20240426 13:20:04.815419 25025 rpcz_store.cc:276] 0426 13:20:02.788274 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.813931 (+2025657us) service_pool.cc:225] Handling call
0426 13:20:04.814237 (+   306us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.814865 (+   628us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.815033 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.817190 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2072) took 2013 ms. Trace:
I20240426 13:20:04.817584 25025 rpcz_store.cc:276] 0426 13:20:02.804307 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.816188 (+2011881us) service_pool.cc:225] Handling call
0426 13:20:04.816493 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.816845 (+   352us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.816878 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.816941 (+    63us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:04.816976 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.817035 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.817142 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.820348 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 898) took 9530 ms. Trace:
I20240426 13:20:04.820708 25025 rpcz_store.cc:276] 0426 13:19:55.289799 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.818600 (+9528801us) service_pool.cc:225] Handling call
0426 13:20:04.820265 (+  1665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.828022 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 527) took 2223 ms. Trace:
I20240426 13:20:04.828578 25102 rpcz_store.cc:276] 0426 13:20:02.604396 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.605324 (+   928us) service_pool.cc:225] Handling call
0426 13:20:02.605629 (+   305us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.606052 (+   423us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:02.606457 (+   405us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.616978 (+ 10521us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:02.646494 (+ 29516us) log.cc:842] Serialized 26 byte log entry
0426 13:20:02.646631 (+   137us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:02.646672 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.646738 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.827750 (+2181012us) raft_consensus.cc:1671] finished
0426 13:20:04.827779 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.827957 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4646,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":6022,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":306,"thread_start_us":10010,"threads_started":1,"wal-append.queue_time_us":99405}
W20240426 13:20:04.833098 24996 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.984s (SENT)
I20240426 13:20:04.833647 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:04.836297 27808 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:20:04.831671 27833 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.032s	user 0.010s	sys 0.010s
I20240426 13:20:04.838423 27833 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.839825 25030 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.842691 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2054) took 4968 ms. Trace:
I20240426 13:20:04.843055 25030 rpcz_store.cc:276] 0426 13:19:59.873952 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.761711 (+4887759us) service_pool.cc:225] Handling call
0426 13:20:04.762436 (+   725us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.840255 (+ 77819us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.842363 (+  2108us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.842595 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177541888}
W20240426 13:20:04.846675 24996 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 2.135s (SENT)
I20240426 13:20:04.855610 27833 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:04.846915 25022 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.044s	user 0.016s	sys 0.015s
I20240426 13:20:04.856475 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:04.856904 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:04.857069 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2061) took 2324 ms (client timeout 1848 ms). Trace:
W20240426 13:20:04.857511 25022 rpcz_store.cc:269] 0426 13:20:02.532511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.691994 (+159483us) service_pool.cc:225] Handling call
0426 13:20:04.857000 (+2165006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.860881 25099 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:04.862046 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.862738 25099 raft_consensus.cc:2362] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4. Re-sending same reply.
I20240426 13:20:04.863669 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.877492 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.877771 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 382) took 6193 ms. Trace:
I20240426 13:20:04.878232 25031 rpcz_store.cc:276] 0426 13:19:58.684491 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.572939 (+5888448us) service_pool.cc:225] Handling call
0426 13:20:04.666150 (+ 93211us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.667361 (+  1211us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.877554 (+210193us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b54004169d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.877693 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213446912}
I20240426 13:20:04.880972 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:04.881686 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0888649749998649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:20:04.881765 27725 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.881944 27808 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
W20240426 13:20:04.882257 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0573649749999277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20240426 13:20:04.882726 27808 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:04.882498 27725 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.883872 27808 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.884738 27725 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.883678 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:04.887188 27725 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:04.887601 27725 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:04.887842 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.888053 27725 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:04.878810 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.912946 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2004) took 10134 ms. Trace:
I20240426 13:20:04.913379 25028 rpcz_store.cc:276] 0426 13:19:54.778183 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.379732 (+1601549us) service_pool.cc:225] Handling call
0426 13:20:02.276248 (+5896516us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.380387 (+104139us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.912263 (+2531876us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.912744 (+   481us) spinlock_profiling.cc:243] Waited 5.87 s on lock 0x7b5400242550. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.912868 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13727666816}
I20240426 13:20:04.889070 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.915273 27630 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.915983 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1710) took 2088 ms. Trace:
I20240426 13:20:04.916435 25105 rpcz_store.cc:276] 0426 13:20:02.828014 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.828156 (+   142us) service_pool.cc:225] Handling call
0426 13:20:04.915887 (+2087731us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.917500 27707 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:04.917814 27911 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:04.917559 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.918908 27911 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.919343 27911 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.891914 27943 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:04.920609 25102 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:04.895315 27708 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.899121 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 510) took 5994 ms. Trace:
I20240426 13:20:04.871338 27833 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:04.921528 27708 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:04.921473 25104 rpcz_store.cc:276] 0426 13:19:58.904992 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.944441 (+ 39449us) service_pool.cc:225] Handling call
0426 13:19:58.945928 (+  1487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.997074 (+ 51146us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:59.395389 (+398315us) log.cc:842] Serialized 14 byte log entry
0426 13:19:59.583331 (+187942us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:59.583379 (+    48us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:59.583423 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.898639 (+5315216us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.898705 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.898921 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":223749,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":235,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":20037,"spinlock_wait_cycles":117074816,"thread_start_us":585608,"threads_started":2,"wal-append.queue_time_us":298248,"wal-append.run_cpu_time_us":11284,"wal-append.run_wall_time_us":1242894}
I20240426 13:20:04.923913 25024 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:04.924613 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1518) took 6317 ms. Trace:
I20240426 13:20:04.924937 25024 rpcz_store.cc:276] 0426 13:19:58.607628 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.923332 (+6315704us) service_pool.cc:225] Handling call
0426 13:20:04.923721 (+   389us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.924387 (+   666us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.924564 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.923772 27708 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.925019 27911 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.929311 27911 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.918182 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.933243 27685 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.934247 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1942) took 5978 ms. Trace:
I20240426 13:20:04.933856 27685 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.934654 24951 rpcz_store.cc:276] 0426 13:19:58.955723 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.955896 (+   173us) service_pool.cc:225] Handling call
0426 13:19:59.263805 (+307909us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:59.291369 (+ 27564us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.933884 (+5642515us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54003c37d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.934070 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707353472}
I20240426 13:20:04.919631 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.936193 25616 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:04.936934 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.937142 27787 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:04.937462 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 615) took 6571 ms. Trace:
I20240426 13:20:04.937628 25104 raft_consensus.cc:2362] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
I20240426 13:20:04.937791 27787 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.938623 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.938987 27725 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.937906 25620 rpcz_store.cc:276] 0426 13:19:58.366445 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.594528 (+228083us) service_pool.cc:225] Handling call
0426 13:19:58.599727 (+  5199us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:58.865940 (+266213us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:58.894253 (+ 28313us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.716675 (+3822422us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.828767 (+112092us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.313066 (+1484299us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.313276 (+   210us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.313328 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.937076 (+623748us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.937131 (+    55us) raft_consensus.cc:1671] finished
0426 13:20:04.937145 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.937300 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3408969,"prepare.queue_time_us":3351321,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":88,"raft.queue_time_us":234215,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":2672,"thread_start_us":713731,"threads_started":4,"wal-append.queue_time_us":997696,"wal-append.run_cpu_time_us":1441,"wal-append.run_wall_time_us":1003520}
W20240426 13:20:04.943086 25621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.162s	user 0.017s	sys 0.025s
W20240426 13:20:04.949126 25621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.168s	user 0.018s	sys 0.026s
I20240426 13:20:04.943722 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.919660 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:04.946139 24995 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.509s (SENT)
I20240426 13:20:04.949945 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:04.950237 27725 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.950397 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.951146 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:04.951700 27855 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.952342 27911 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:20:04.952543 25104 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 13:20:04.952582 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.953166 25104 pending_rounds.cc:76] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 2
I20240426 13:20:04.953487 25104 pending_rounds.cc:97] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 13:20:04.953763 25104 raft_consensus.cc:2866] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.954308 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.955282 27451 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.955948 27451 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:04.957026 25104 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240426 13:20:04.957504 25104 pending_rounds.cc:76] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 4
I20240426 13:20:04.957808 25104 pending_rounds.cc:97] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240426 13:20:04.959317 25104 raft_consensus.cc:2866] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.965900 25104 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:04.966703 27451 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:04.965759 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.968475 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.969146 27451 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.972440 27725 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:04.973426 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:04.974117 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:04.974725 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 584) took 2155 ms (client timeout 1716 ms). Trace:
I20240426 13:20:04.979532 27451 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:20:04.975306 25618 rpcz_store.cc:269] 0426 13:20:02.818848 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.820929 (+  2081us) service_pool.cc:225] Handling call
0426 13:20:04.973262 (+2152333us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:04.974629 (+  1367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15843072}
I20240426 13:20:04.981367 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.980237 27725 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.982381 27911 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:04.983919 27911 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:04.985105 27868 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.984983 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.985548 27868 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.985677 27911 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:04.986675 27868 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.986999 27376 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:04.987350 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.987651 27376 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.987951 27868 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.988233 27911 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:04.988634 27376 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.989761 27911 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:04.990017 27376 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:04.990875 27911 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.991274 27911 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.992376 27833 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.992949 27833 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.987093 27923 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:04.994728 25026 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.264s	user 0.020s	sys 0.002s
I20240426 13:20:04.995088 27833 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.995115 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:04.995734 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.996866 27633 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.997346 27633 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:04.998508 27986 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.414s	user 0.006s	sys 0.000s
I20240426 13:20:05.001477 27633 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:04.992373 27911 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.003137 27911 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.003753 27736 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.004901 27736 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:20:05.010109 24995 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.135s (SENT)
I20240426 13:20:05.010586 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.011432 27633 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240426 13:20:05.018030 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1954) took 4737 ms (client timeout 1926 ms). Trace:
W20240426 13:20:05.018489 24956 rpcz_store.cc:269] 0426 13:20:00.280397 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.280519 (+   122us) service_pool.cc:225] Handling call
0426 13:20:05.017691 (+4737172us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b54000a7dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.017831 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6806400}
I20240426 13:20:05.021842 27685 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:05.026795 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 174) took 23929 ms (client timeout 15000 ms). Trace:
W20240426 13:20:05.028889 24853 rpcz_store.cc:269] 0426 13:19:41.097084 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.097474 (+   390us) service_pool.cc:225] Handling call
0426 13:20:05.026581 (+23929107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:52.929618 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:53.108553 (+178935us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:53.108614 (+    61us) write_op.cc:427] Acquired schema lock
0426 13:19:53.108634 (+    20us) tablet.cc:660] Decoding operations
0426 13:19:53.109594 (+   960us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:53.109691 (+    97us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:53.109721 (+    30us) tablet.cc:683] Acquiring locks for 7 operations
0426 13:19:53.110452 (+   731us) tablet.cc:699] Row locks acquired
0426 13:19:53.110491 (+    39us) write_op.cc:260] PREPARE: finished
0426 13:19:53.110681 (+   190us) write_op.cc:270] Start()
0426 13:19:53.110821 (+   140us) write_op.cc:275] Timestamp: P: 1714137593110616 usec, L: 0
0426 13:19:53.110853 (+    32us) op_driver.cc:331] REPLICATION: starting
0426 13:19:53.111549 (+   696us) log.cc:842] Serialized 2793 byte log entry
0426 13:19:53.902042 (+790493us) op_driver.cc:444] REPLICATION: finished
0426 13:19:54.253470 (+351428us) write_op.cc:300] APPLY: starting
0426 13:19:54.303504 (+ 50034us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:54.341491 (+ 37987us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:54.399281 (+ 57790us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:54.868470 (+469189us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:54.871042 (+  2572us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0426 13:19:54.871111 (+    69us) write_op.cc:311] APPLY: finished
0426 13:19:54.871482 (+   371us) log.cc:842] Serialized 59 byte log entry
0426 13:19:55.021765 (+150283us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:55.036844 (+ 15079us) write_op.cc:446] Released schema lock
0426 13:19:55.051202 (+ 14358us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:55.079513 (+ 28311us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:55.137570 (+ 58057us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":283525,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":289210,"apply.run_cpu_time_us":27283,"apply.run_wall_time_us":939705,"num_ops":7,"prepare.queue_time_us":832991,"prepare.run_cpu_time_us":17746,"prepare.run_wall_time_us":288630,"raft.queue_time_us":307879,"raft.run_cpu_time_us":63993,"raft.run_wall_time_us":236014,"replication_time_us":791089,"thread_start_us":1302722,"threads_started":5,"wal-append.queue_time_us":539770,"wal-append.run_cpu_time_us":23292,"wal-append.run_wall_time_us":2532056}]]}
I20240426 13:20:05.037691 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.039086 27755 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:05.039484 27755 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.039820 27755 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:05.045380 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55921.6us
I20240426 13:20:05.044438 27932 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:05.048107 27633 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:05.052857 25096 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:05.061383 27955 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:05.071065 25622 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:05.071652 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.072335 25104 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:05.073088 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:05.073673 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 585) took 2254 ms (client timeout 1824 ms). Trace:
I20240426 13:20:05.074052 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.074853 25104 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:05.077968 27974 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.078395 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.078578 27974 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.080214 27633 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:05.080678 27633 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.081028 27633 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.081406 25027 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:05.082043 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1515) took 6635 ms. Trace:
I20240426 13:20:05.082355 25027 rpcz_store.cc:276] 0426 13:19:58.446335 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.080820 (+6634485us) service_pool.cc:225] Handling call
0426 13:20:05.081204 (+   384us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.081804 (+   600us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.081980 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:05.074085 25622 rpcz_store.cc:269] 0426 13:20:02.819637 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.819776 (+   139us) service_pool.cc:225] Handling call
0426 13:20:05.073513 (+2253737us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.073609 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5167418752}
W20240426 13:20:05.096761 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 619) took 5563 ms (client timeout 2123 ms). Trace:
W20240426 13:20:05.097301 25616 rpcz_store.cc:269] 0426 13:19:59.532913 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.620244 (+ 87331us) service_pool.cc:225] Handling call
0426 13:20:05.096490 (+5476246us) spinlock_profiling.cc:243] Waited 5.29 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.096678 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12177081088}
I20240426 13:20:05.101161 27787 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:05.099421 27633 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.102736 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.126399 27869 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:05.127151 27974 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.127782 27869 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.128206 27869 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.129089 27869 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.138463 25029 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:05.144704 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.145547 25096 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.146926 25103 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:05.147310 25103 pending_rounds.cc:76] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:20:05.147568 25103 pending_rounds.cc:97] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:05.147835 25103 raft_consensus.cc:2866] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:05.148147 26679 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240426 13:20:05.151104 25069 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.926s (SENT)
I20240426 13:20:05.150902 26679 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 13:20:05.152889 25070 connection.cc:232] RPC call timeout handler was delayed by -1.6116430530000798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.209s for a response
W20240426 13:20:05.153472 25070 connection.cc:232] RPC call timeout handler was delayed by -1.4123543020000398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18480000000000002s for a response
W20240426 13:20:05.153867 25070 connection.cc:232] RPC call timeout handler was delayed by -1.3377743649999729s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20240426 13:20:05.154258 25070 connection.cc:232] RPC call timeout handler was delayed by -1.1357782520000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
I20240426 13:20:05.125743 27755 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.161828 27808 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:05.162470 27808 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.164171 27943 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.164130 27932 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:05.137924 27633 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.168792 25103 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:20:05.170583 27755 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.163000 27855 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.164649 27808 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.181540 27855 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.183615 27855 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.164827 27943 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.185623 27943 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.186586 27911 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:05.186965 27911 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.187867 27911 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.189056 27911 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:05.190026 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1513) took 6964 ms (client timeout 2473 ms). Trace:
W20240426 13:20:05.190554 25026 rpcz_store.cc:269] 0426 13:19:58.225260 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.346301 (+121041us) service_pool.cc:225] Handling call
0426 13:20:05.189732 (+6843431us) spinlock_profiling.cc:243] Waited 4.02 s on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.189932 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9252995328}
I20240426 13:20:05.193713 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.196517 25104 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.180074 27869 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.203752 27755 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
W20240426 13:20:05.227371 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60767.3us
I20240426 13:20:05.228612 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.233186 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.239099 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.242851 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.243352 25026 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.246198 27946 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:05.246639 25024 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.247502 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.248605 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.249548 26679 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:05.250053 26679 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:05.250113 27978 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.549s	user 0.001s	sys 0.002s
I20240426 13:20:05.250960 26679 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.252249 27946 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:05.256068 27868 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:05.247799 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.255885 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.260648 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:05.261134 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 583) took 2443 ms (client timeout 2047 ms). Trace:
W20240426 13:20:05.261569 25617 rpcz_store.cc:269] 0426 13:20:02.818203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.820748 (+  2545us) service_pool.cc:225] Handling call
0426 13:20:05.255583 (+2434835us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:05.261041 (+  5458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3658022784}
I20240426 13:20:05.262054 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2043) took 5866 ms. Trace:
I20240426 13:20:05.262516 25029 rpcz_store.cc:276] 0426 13:19:59.395940 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.719847 (+5323907us) service_pool.cc:225] Handling call
0426 13:20:04.728916 (+  9069us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.139931 (+411015us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.261792 (+121861us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:05.261978 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":941240704}
I20240426 13:20:05.263521 27653 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:05.264079 27653 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.264961 27653 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.266865 27376 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.268105 27376 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:05.268487 27376 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.269366 27376 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.270499 27376 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.275759 25100 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:05.258952 27911 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.284905 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.286453 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.296551 25026 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:05.297653 25029 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.302390 24852 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.303488 27725 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:20:05.305085 27725 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.305262 24852 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:05.309536 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.310321 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.310428 27725 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.310938 25098 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:05.311497 27498 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:05.312016 27498 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.313030 27498 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.316592 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.317559 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.318781 27755 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:05.319177 27755 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.319506 27755 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:05.311827 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.322110 25101 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:05.324354 27633 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:05.325430 27633 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:05.287842 27911 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.325830 27633 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.330423 27653 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:05.332816 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 520) took 2946 ms. Trace:
I20240426 13:20:05.333240 25100 rpcz_store.cc:276] 0426 13:20:02.387429 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.409006 (+ 21577us) service_pool.cc:225] Handling call
0426 13:20:02.412809 (+  3803us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.276203 (+2863394us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.332600 (+ 56397us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:05.332761 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6583452544}
I20240426 13:20:05.330220 27633 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.337649 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.332687 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.340590 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.338274 25096 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:05.307894 24852 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.341905 27633 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.341753 27725 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:05.346457 27376 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:05.346628 27725 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.347061 27376 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.347528 27376 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.347775 27725 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.327239 27911 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.352792 27911 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.352859 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.353574 25031 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:05.356918 27930 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.357472 27930 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.357487 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.294128 27916 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.363126 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.363895 25616 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.363536 24852 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:05.365207 24852 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.367417 27808 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:05.367985 27808 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.370411 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.369985 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.371362 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.369038 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.369551 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.377697 25622 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.367188 24852 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:05.387913 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91876.4us
I20240426 13:20:05.371060 27808 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.390460 27376 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.394599 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.395898 25620 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.369552 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.397797 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.423811 25623 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:05.400329 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.400439 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.425230 25024 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.425733 25023 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:05.384675 24852 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.422791 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.427739 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.427073 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 415) took 4829 ms. Trace:
I20240426 13:20:05.380806 27916 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.429246 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.430850 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2023) took 6822 ms. Trace:
I20240426 13:20:05.432358 27438 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:05.432730 27955 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.432686 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 27) took 2922 ms. Trace:
I20240426 13:20:05.381940 27930 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.430423 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.435562 25023 rpcz_store.cc:276] 0426 13:19:58.608194 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.392200 (+6784006us) service_pool.cc:225] Handling call
0426 13:20:05.418700 (+ 26500us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.427684 (+  8984us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.430780 (+  3096us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:05.437548 25097 rpcz_store.cc:276] 0426 13:20:02.504248 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.504744 (+   496us) service_pool.cc:225] Handling call
0426 13:20:05.423918 (+2919174us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.424026 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577234176}
I20240426 13:20:05.441754 25104 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:05.443009 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.444612 27932 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:05.443586 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:05.452867 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.433465 27946 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.453737 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.476739 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.445088 27932 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.486217 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.449548 27438 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.452457 27498 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:05.451961 27955 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.429831 25615 rpcz_store.cc:276] 0426 13:19:58.999038 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.999183 (+   145us) service_pool.cc:225] Handling call
0426 13:19:58.999556 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.999770 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:59.000132 (+   362us) log.cc:842] Serialized 14 byte log entry
0426 13:19:59.211809 (+211677us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.716114 (+3504305us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda941c17a8 00007fda8d3ecff6 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34
0426 13:20:02.769975 (+ 53861us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:03.158454 (+388479us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:03.176169 (+ 17715us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:03.310599 (+134430us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:03.348324 (+ 37725us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:03.828055 (+479731us) inbound_call.cc:173] Queueing success response
0426 13:20:04.704986 (+876931us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":52914,"prepare.queue_time_us":27128,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":191214,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":13045,"spinlock_wait_cycles":61451776,"thread_start_us":211881,"threads_started":2,"wal-append.queue_time_us":391092,"wal-append.run_cpu_time_us":5950,"wal-append.run_wall_time_us":1230833}
I20240426 13:20:05.495637 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.461103 27946 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.501936 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.502218 27438 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.503094 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.503166 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:05.504405 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 5.22676e+06us
I20240426 13:20:05.505574 27376 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.489594 27932 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.496659 27498 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:05.507689 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080003bae0 after lost signal to thread 23799
W20240426 13:20:05.522996 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003984e0 after lost signal to thread 24888
W20240426 13:20:05.523711 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080034e5e0 after lost signal to thread 24920
I20240426 13:20:05.539324 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:05.538980 24852 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.540700 24852 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:05.543195 25024 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:05.536971 27498 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.548542 27955 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.550396 25030 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:05.550688 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.555608 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.557159 25023 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.497879 27810 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20240426 13:20:05.561286 27794 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:05.570310 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 207616us
W20240426 13:20:05.577256 28007 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:20:05.586108 27653 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:05.586558 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:05.586778 27653 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:05.588348 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800333600 after lost signal to thread 24980
I20240426 13:20:05.587796 27653 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.588671 27310 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:05.589591 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800407220 after lost signal to thread 25055
I20240426 13:20:05.536711 27916 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.590894 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003a8a80 after lost signal to thread 25129
I20240426 13:20:05.591321 27696 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:05.591786 27696 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:05.569316 27946 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.593580 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003ef3a0 after lost signal to thread 25650
W20240426 13:20:05.594512 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080033f000 after lost signal to thread 27631
I20240426 13:20:05.589128 27310 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:05.597186 27946 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:05.597666 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08000395c0 after lost signal to thread 27640
W20240426 13:20:05.602329 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003eeae0 after lost signal to thread 27684
W20240426 13:20:05.602563 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080029f020 after lost signal to thread 27686
W20240426 13:20:05.597507 27376 consensus_queue.cc:1159] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 7 status { last_received { term: 6 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:20:05.602726 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080030f180 after lost signal to thread 27687
W20240426 13:20:05.604730 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7180 after lost signal to thread 27710
I20240426 13:20:05.606659 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.609462 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.607404 25029 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:20:05.622882 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800333ae0 after lost signal to thread 27717
I20240426 13:20:05.624084 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:20:05.624541 25589 connection.cc:232] RPC call timeout handler was delayed by -1.2860821720000786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20240426 13:20:05.626067 25589 connection.cc:232] RPC call timeout handler was delayed by -1.1427117030000318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20240426 13:20:05.629307 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:05.630040 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.632172 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.639016 26679 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:05.639384 26679 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.639591 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.639647 26679 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:05.644680 25104 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:05.649919 25031 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:05.604562 27755 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.642468 27833 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:05.592666 27696 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.642606 27725 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240426 13:20:05.650455 27868 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.702587 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.703637 25096 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:05.703761 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.704463 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:05.706326 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.664060 25616 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20240426 13:20:05.664363 27310 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:05.664762 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.708513 25617 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.667510 27410 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:05.668320 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.716696 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.699708 27868 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.718140 27868 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.670718 27755 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.674782 25105 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:05.667495 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 416) took 5385 ms. Trace:
I20240426 13:20:05.720525 25099 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:05.720990 25099 raft_consensus.cc:603] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:05.720475 25621 rpcz_store.cc:276] 0426 13:20:00.282461 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.283457 (+   996us) service_pool.cc:225] Handling call
0426 13:20:00.283802 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:00.284036 (+   234us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:02.780848 (+2496812us) log.cc:842] Serialized 14 byte log entry
0426 13:20:05.652161 (+2871313us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:05.659751 (+  7590us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:05.667183 (+  7432us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.667283 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:05.667436 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3126256,"prepare.queue_time_us":2433729,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":694615,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":6070,"thread_start_us":2227457,"threads_started":2,"wal-append.queue_time_us":1711245,"wal-append.run_cpu_time_us":1743,"wal-append.run_wall_time_us":1016151}
I20240426 13:20:05.687990 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.694236 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.727073 25098 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:20:05.727108 25100 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20240426 13:20:05.698642 27932 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.732781 25100 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:20:05.741294 25100 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:05.741643 25100 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:05.743767 25098 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:05.747489 25023 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:05.749511 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:05.760416 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:05.773028 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146093us
I20240426 13:20:05.772569 27869 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:05.801821 27869 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.806922 25023 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.802788 26679 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.794723 27794 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:05.838176 27498 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.869966 28019 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20240426 13:20:05.927487 28014 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.369s	user 0.002s	sys 0.002s
I20240426 13:20:05.975881 25104 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 4. Ops: []
I20240426 13:20:05.976931 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 587) took 3150 ms. Trace:
I20240426 13:20:05.988502 25624 rpcz_store.cc:276] 0426 13:20:02.826410 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.826658 (+   248us) service_pool.cc:225] Handling call
0426 13:20:04.853461 (+2026803us) spinlock_profiling.cc:243] Waited 871 us on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:04.853657 (+   196us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.603300 (+749643us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:05.603340 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:05.603832 (+   492us) log.cc:842] Serialized 26 byte log entry
0426 13:20:05.784734 (+180902us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:05.784774 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.784851 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:05.976599 (+191748us) raft_consensus.cc:1671] finished
0426 13:20:05.976619 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:05.976873 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":749199,"raft.queue_time_us":749835,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":2004352,"thread_start_us":180469,"threads_started":1,"wal-append.queue_time_us":206095}
I20240426 13:20:05.743674 27725 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
I20240426 13:20:05.991575 28035 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:20:05.991880 25025 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:05.991465 27708 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:05.993003 27708 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:20:05.802781 27869 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.979784 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.995262 25622 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:05.996326 27932 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.003371 25622 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.819640 27916 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20240426 13:20:06.004671 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:05.983271 26679 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.808017 27955 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:20:06.005800 26679 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:05.994001 28035 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.009017 27794 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:06.010110 27794 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.006297 26679 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:06.014417 26679 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.020117 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:06.020812 25621 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.023114 25098 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:06.025172 27932 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:06.031261 25621 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:06.032483 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:06.033182 27498 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.033525 27932 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:06.034005 25098 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:20:06.034735 28033 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:20:06.040607 27749 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.041373 27766 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.041241 27749 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.041968 27766 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.043334 27749 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.043746 27766 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.043934 27749 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.057123 27775 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:06.044153 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800113620 after lost signal to thread 27753
W20240426 13:20:06.068588 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800344bc0 after lost signal to thread 27756
W20240426 13:20:06.068979 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080035caa0 after lost signal to thread 27768
W20240426 13:20:06.069301 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800339a80 after lost signal to thread 27771
I20240426 13:20:06.068055 27775 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.052662 27881 raft_consensus.cc:659] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } [suppressed 5 similar messages]
I20240426 13:20:06.069562 27410 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:06.070355 27410 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.067677 27749 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.072126 27928 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.072582 27749 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:06.073475 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080040a280 after lost signal to thread 27811
W20240426 13:20:06.073836 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800249900 after lost signal to thread 27813
W20240426 13:20:06.074465 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003e2dc0 after lost signal to thread 27838
I20240426 13:20:06.073513 27928 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.074726 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003341c0 after lost signal to thread 27841
W20240426 13:20:06.075383 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080039bea0 after lost signal to thread 27844
W20240426 13:20:06.075719 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4aa0 after lost signal to thread 27849
I20240426 13:20:06.076198 27900 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.076541 28036 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:06.076961 28036 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:06.076678 27900 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.051348 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1728) took 1189 ms. Trace:
I20240426 13:20:06.081954 25105 rpcz_store.cc:276] 0426 13:20:04.862125 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.369293 (+507168us) service_pool.cc:225] Handling call
0426 13:20:06.038290 (+668997us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:06.051254 (+ 12964us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678655744}
W20240426 13:20:06.141609 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 331370us
I20240426 13:20:06.063899 25621 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.146435 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:06.115307 28026 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20240426 13:20:06.055716 27833 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:06.147852 25100 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.153602 27770 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.154163 27770 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.156296 28036 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.157714 28036 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.991484 25617 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.147361 25621 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:06.161648 27833 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:06.145128 27932 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:06.139770 27755 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20240426 13:20:06.181589 25022 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:06.175024 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:06.191511 24956 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20240426 13:20:06.191586 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.192168 27730 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:06.192641 27730 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.193002 27730 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.193279 27775 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.166242 27410 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.194487 25621 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.195364 27410 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.183265 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.196154 27637 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.196280 24947 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.196736 27637 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:06.198310 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49040.4us
I20240426 13:20:06.198805 27637 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.186604 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.187315 27900 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.200860 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 13:20:06.202986 27736 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:06.203428 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:20:06.204151 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:06.204319 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 665065us
I20240426 13:20:06.204831 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.186515 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.193104 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.188143 27310 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:06.188273 24920 connection.cc:232] RPC call timeout handler was delayed by -4.73400823999998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
I20240426 13:20:06.205812 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:06.171640 27770 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.209254 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:06.219538 25100 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:06.223131 24947 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:06.206110 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.195835 25621 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.221372 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.205770 27736 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.241163 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.186688 28025 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.240465 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.245276 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.252454 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:06.250501 24952 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:06.272585 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66851.1us
I20240426 13:20:06.207983 27647 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.279875 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:06.282709 27759 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:06.282812 27932 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.283799 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:06.284557 25023 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.289134 25031 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:20:06.285771 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:06.284765 27810 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:06.292531 25026 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:06.292461 27810 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.293866 25025 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:06.294505 27810 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.277638 27725 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20240426 13:20:06.296344 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.298688 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:06.245793 27770 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.299849 27914 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:06.220561 27964 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20240426 13:20:06.240664 27736 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:06.301548 27410 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:06.302557 27736 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.303745 27736 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.285768 27310 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.273473 27853 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:06.300225 27881 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:06.284876 25622 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:20:06.304126 27647 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.304600 27914 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:06.305445 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.559173 24949 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:20:06.304284 27410 consensus_queue.cc:1159] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 6 status { last_received { term: 6 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 6 index: 8 } }
I20240426 13:20:06.306015 27271 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:06.307265 27880 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:06.308751 24947 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:06.443259 27382 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.093s
I20240426 13:20:06.309973 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.310645 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.311693 24956 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:20:06.300262 27770 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.314159 28025 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.415686 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:06.575459 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 159856us
W20240426 13:20:06.403496 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91085us
I20240426 13:20:06.413038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.480723 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:06.576352 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:06.326064 27881 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:06.421896 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 105090us
I20240426 13:20:06.407497 25031 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:06.396458 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:06.412212 27725 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.232981 27451 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.057s
I20240426 13:20:06.301587 27964 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:06.325775 27853 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:06.443246 27653 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:06.443691 26679 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.441848 27350 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
W20240426 13:20:06.412366 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 121347us
I20240426 13:20:06.597591 27881 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.309334 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.660992 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:06.580642 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.568142 24950 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:06.583163 27946 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.576867 24955 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.495672 27730 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.627753 24953 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.627264 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:06.628185 25618 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:20:06.628832 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.632158 27757 raft_consensus.cc:2741] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:20:06.583736 27923 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:06.608628 27964 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:06.597404 28027 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20240426 13:20:06.567343 27916 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.618085 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.651072 26679 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.403272 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.656976 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.638303 27853 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:06.664371 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56021us
I20240426 13:20:06.664681 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.662257 24947 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.668278 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:06.675928 25623 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:20:06.637780 28025 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.677511 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:06.677778 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 456434us
I20240426 13:20:06.678263 24949 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:20:06.686618 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.035s	sys 0.047s
I20240426 13:20:06.609207 27725 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.586398 27271 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.672318 27946 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.692363 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:06.695809 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.035s	sys 0.047s
I20240426 13:20:06.701265 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.603446 27833 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.705233 24956 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:06.709131 26679 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.645576 27707 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20240426 13:20:06.704747 28044 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.142s	user 0.000s	sys 0.002s
I20240426 13:20:06.707888 24955 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:06.709772 26679 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:20:06.713550 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62891.7us
I20240426 13:20:06.719221 27350 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:06.719610 27350 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:06.670032 27911 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20240426 13:20:06.752347 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.756238 25623 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:06.711803 27630 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.027s	sys 0.057s
I20240426 13:20:06.719749 27647 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.766623 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.760701 27637 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.768923 27633 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.770488 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.772181 27766 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:06.773157 27964 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:06.773941 28037 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.585s	user 0.001s	sys 0.003s
I20240426 13:20:06.720211 27852 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.100s
I20240426 13:20:06.781795 27946 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.708495 27916 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.786082 28041 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20240426 13:20:06.719961 27350 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.788897 25616 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:06.897166 27730 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.791290 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.792562 24950 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.936188 24950 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.793345 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.802044 27725 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.802801 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.804699 27271 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:06.807652 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49452.3us
I20240426 13:20:06.829556 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:06.832739 25097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.021s	sys 0.010s
I20240426 13:20:06.713129 27869 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20240426 13:20:06.809988 27810 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20240426 13:20:06.860411 24995 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.514s (SENT)
W20240426 13:20:06.846199 25621 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.000s
I20240426 13:20:06.893216 24955 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:06.850281 27630 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.029s	sys 0.057s
I20240426 13:20:06.847055 27770 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.892172 27911 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:06.894851 27932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.023s	sys 0.019s
I20240426 13:20:06.790535 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.935678 24949 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.720345 28034 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.259s
W20240426 13:20:06.932606 27808 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.006s	sys 0.032s
I20240426 13:20:06.944152 25618 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:06.939819 24950 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.977262 24950 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:06.946230 25097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.021s	sys 0.010s
I20240426 13:20:06.941371 27946 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.951249 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:06.979686 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119335us
I20240426 13:20:06.980621 24951 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:20:06.951803 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 145083us
I20240426 13:20:06.981073 24951 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.982084 24951 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.983476 24951 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:06.983812 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:06.984385 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.985152 24949 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:06.985374 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:06.986678 24950 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:06.987035 24950 pending_rounds.cc:76] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:06.987238 24950 pending_rounds.cc:97] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:20:06.899173 27633 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.989351 27633 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:06.954993 28045 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:20:06.991760 25103 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:06.899081 27637 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.961638 27932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.023s	sys 0.019s
I20240426 13:20:06.937878 27833 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.054036 27916 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.054941 27310 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.059954 27677 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:07.060524 27677 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:07.061486 27677 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.062647 25624 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:07.063112 25624 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:07.063298 27677 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.064047 25624 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.064714 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.065333 25624 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:07.065726 25103 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:07.066987 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.972846 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.068346 24947 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.927948 27766 consensus_queue.cc:1225] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:06.987183 27900 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:06.991581 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.072746 27900 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.992370 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.073174 27900 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.074259 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.075390 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.993661 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.075953 25030 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.076797 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.076936 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.993903 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.994230 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.077795 25099 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.078348 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.078611 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.078956 25031 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.079253 25099 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:06.994793 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.080802 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.081588 25098 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:07.082293 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.083264 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:06.995018 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.995234 27759 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:06.995720 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.996596 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.996577 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.994259 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.048928 25621 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:06.995632 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.997931 24950 raft_consensus.cc:2866] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.096495 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.001003 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.097441 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.004560 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.013250 28025 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:07.029888 24954 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:06.992370 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.946614 25617 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:06.952877 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 598) took 1258 ms. Trace:
I20240426 13:20:07.039281 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.959764 27730 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:07.112758 25024 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:07.113114 27730 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.113680 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.114318 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:07.114061 27730 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.115171 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.115852 25024 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.116137 27916 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.121105 27916 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.122179 27637 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.123030 27916 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.073096 25023 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.123445 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.124181 24955 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.124291 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.124887 25023 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.124948 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.125763 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.126067 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.126842 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.127105 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.127686 24955 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:07.128232 25023 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
W20240426 13:20:07.131027 28036 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.973s	user 0.011s	sys 0.000s
W20240426 13:20:07.087091 25029 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.000s	sys 0.067s
I20240426 13:20:07.089098 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.131922 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20240426 13:20:07.132165 25099 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:20:07.132283 25104 pending_rounds.cc:76] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 8
I20240426 13:20:07.132620 25104 pending_rounds.cc:97] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20240426 13:20:07.132848 25104 raft_consensus.cc:2866] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.133136 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.133723 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:07.093124 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.093953 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.135075 25101 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.135110 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.136008 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.136834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.095063 25026 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.137239 25101 raft_consensus.cc:1203] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[7.4-7.4]   Dedup: 7.4->[]
I20240426 13:20:07.051376 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:07.139889 25029 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.000s	sys 0.068s
I20240426 13:20:07.141145 25105 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:07.134572 25025 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.116801 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.117616 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.979641 27943 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:07.118611 28035 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.120162 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.077473 25030 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.040572 27946 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.098745 24954 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.137696 25101 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:07.111588 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.089949 27759 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.975147 27911 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.158757 25028 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.136022 24953 raft_consensus.cc:2396] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:07.138960 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.120177 25624 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:07.099558 25615 rpcz_store.cc:276] 0426 13:20:05.687392 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.687788 (+   396us) service_pool.cc:225] Handling call
0426 13:20:05.688133 (+   345us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.809598 (+121465us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:05.809629 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:05.817033 (+  7404us) log.cc:842] Serialized 26 byte log entry
0426 13:20:06.117256 (+300223us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:06.117314 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:06.117405 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:06.773313 (+655908us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:06.945005 (+171692us) raft_consensus.cc:1671] finished
0426 13:20:06.945032 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:06.945295 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":138159,"raft.queue_time_us":121926,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4460,"spinlock_wait_cycles":14870912,"thread_start_us":135554,"threads_started":1,"wal-append.queue_time_us":214630}
I20240426 13:20:07.144770 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.148775 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 492363us
I20240426 13:20:07.155815 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.164494 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.098575 28025 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.240782 28025 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.240689 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.242094 28025 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.242321 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:07.243216 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.243783 25027 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:07.091172 27881 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.244519 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.245079 25027 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:07.245779 27881 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.246300 27725 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:07.246942 27725 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:07.247809 27725 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.248521 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.249395 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.250159 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.196292 27943 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.156899 28035 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.208472 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.209385 27653 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.215471 25105 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:07.216430 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.218343 27451 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.154685 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 602) took 1091 ms. Trace:
I20240426 13:20:07.274113 24953 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.236660 27946 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.237329 28036 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20240426 13:20:07.164193 27916 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.252077 25027 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:07.165900 24954 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.262164 25096 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.273872 27869 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.215654 24949 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:07.302824 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.303550 24949 raft_consensus.cc:2362] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
W20240426 13:20:07.292069 24956 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.000s
I20240426 13:20:07.252755 27647 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:07.255112 25100 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:07.305648 25031 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.297930 24954 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.299388 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.300475 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.263525 27943 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.305485 24956 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:07.309280 24955 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.310585 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.036s	sys 0.049s
I20240426 13:20:07.312119 24953 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:07.312105 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1994) took 1004 ms. Trace:
I20240426 13:20:07.315244 24956 rpcz_store.cc:276] 0426 13:20:06.307242 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.589369 (+282127us) service_pool.cc:225] Handling call
0426 13:20:07.312024 (+722655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:07.316526 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.316731 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:07.317772 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.037s	sys 0.049s
I20240426 13:20:07.304477 27271 raft_consensus.cc:2741] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20240426 13:20:07.300637 27946 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.203920 27637 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.098s
I20240426 13:20:07.318915 24956 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.317656 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.302182 27869 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:07.324894 25588 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.752s (SENT) [suppressed 42 similar messages]
I20240426 13:20:07.324412 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.328553 24949 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.319226 27943 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.306349 27647 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.351254 24953 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:07.293874 27653 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.294056 27451 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.353181 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.027s	sys 0.017s
I20240426 13:20:07.289695 25621 rpcz_store.cc:276] 0426 13:20:06.063115 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.063286 (+   171us) service_pool.cc:225] Handling call
0426 13:20:06.063732 (+   446us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.095356 (+1031624us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.150110 (+ 54754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:07.325914 27946 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.363692 24956 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:07.361788 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.308715 27755 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20240426 13:20:07.361851 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.027s	sys 0.018s
I20240426 13:20:07.330984 27869 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.307807 27916 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.391448 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 696476us
I20240426 13:20:07.405082 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.363015 27382 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.033s	sys 0.018s
W20240426 13:20:07.360311 27928 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.017s	sys 0.027s
W20240426 13:20:07.363440 27708 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.014s	sys 0.037s
W20240426 13:20:07.413455 27868 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.046s	sys 0.046s
I20240426 13:20:07.420269 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.389602 27946 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.363631 27633 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.302385 27759 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.425172 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.014s	sys 0.079s
I20240426 13:20:07.412431 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:07.425776 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.440822 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.015s	sys 0.080s
W20240426 13:20:07.441175 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 843699us
I20240426 13:20:07.428130 25099 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:20:07.413918 27382 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.034s	sys 0.018s
I20240426 13:20:07.441815 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.454239 26679 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:07.422899 27928 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.017s	sys 0.027s
W20240426 13:20:07.424082 27708 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.015s	sys 0.037s
I20240426 13:20:07.459537 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.424695 27868 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.047s	sys 0.046s
I20240426 13:20:07.451602 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.461503 26679 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.391541 28036 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:20:07.497994 26679 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.502125 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.460465 27914 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:07.509696 25100 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:07.460122 28034 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:07.481550 27911 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:07.513084 24948 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.029s	sys 0.025s
W20240426 13:20:07.514889 24948 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.030s	sys 0.025s
I20240426 13:20:07.501626 27759 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:07.521831 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.524888 24953 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:07.526717 25100 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:07.528357 28050 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20240426 13:20:07.481245 27647 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.516865 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:07.537942 27869 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.513059 27853 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.805s	user 0.008s	sys 0.004s
I20240426 13:20:07.544430 27853 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.546838 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.014s	sys 0.080s
W20240426 13:20:07.547253 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.014s	sys 0.080s
I20240426 13:20:07.548167 27853 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.449855 27633 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.594923 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.595840 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:07.552201 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44737.3us
I20240426 13:20:07.597632 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.598954 27271 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.599463 27271 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.599874 27271 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.601785 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84838.3us
I20240426 13:20:07.549886 28035 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.619189 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.620085 25100 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:07.630492 24951 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.000s
I20240426 13:20:07.636766 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.523905 27382 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20240426 13:20:07.650687 25028 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:07.600268 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:07.664662 25105 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
W20240426 13:20:07.670344 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.000s	sys 0.016s
I20240426 13:20:07.670424 27869 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.671192 27911 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:20:07.673687 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.673867 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1791) took 1432 ms. Trace:
I20240426 13:20:07.674458 27911 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:07.675024 27911 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.675725 27911 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.674376 25097 rpcz_store.cc:276] 0426 13:20:06.240962 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.241183 (+   221us) service_pool.cc:225] Handling call
0426 13:20:06.285283 (+ 44100us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:06.310969 (+ 25686us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:06.311004 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:06.311470 (+   466us) log.cc:842] Serialized 26 byte log entry
0426 13:20:06.978294 (+666824us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:06.978331 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:06.978394 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:07.673566 (+695172us) raft_consensus.cc:1671] finished
0426 13:20:07.673601 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:07.673775 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1209,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":666458,"threads_started":1,"wal-append.queue_time_us":1324373}
I20240426 13:20:07.677500 25097 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:07.662461 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:07.701272 24954 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:07.679459 27943 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.732674 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.697003 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:07.716565 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.012s	sys 0.012s
W20240426 13:20:07.736732 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.012s	sys 0.012s
I20240426 13:20:07.717504 24955 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.714323 24951 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:07.727238 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.740939 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.670778 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.000s	sys 0.016s
I20240426 13:20:07.656800 27451 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.743714 24947 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:20:07.744125 24947 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:07.744344 24947 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:20:07.744488 25105 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:20:07.744642 24947 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.740928 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.727869 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54262.1us
I20240426 13:20:07.753212 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.754299 25024 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.699465 27653 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:07.756086 24995 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.013s	sys 0.024s
I20240426 13:20:07.756321 24954 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:20:07.707265 27916 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.737521 27869 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:07.737663 28044 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.015s	sys 0.007s
W20240426 13:20:07.780344 24995 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.015s	sys 0.024s
W20240426 13:20:07.784771 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 803799us
I20240426 13:20:07.786317 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.721949 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:07.788681 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:07.782707 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.669643 28061 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:07.679816 27923 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:07.798236 27923 consensus_queue.cc:1225] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:07.773898 27964 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:20:07.754717 27808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.016s	sys 0.025s
I20240426 13:20:07.762125 28034 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:07.784380 28044 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.016s	sys 0.008s
W20240426 13:20:07.799952 24995 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.683s (SENT)
I20240426 13:20:07.800499 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.815639 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.815294 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:07.819723 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.774969 27451 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.847359 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.849807 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.851362 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51450us
W20240426 13:20:07.852324 24995 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.704s (SENT)
I20240426 13:20:07.853147 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:07.850668 24955 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:20:07.808441 27808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.016s	sys 0.025s
I20240426 13:20:07.856859 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.859236 25100 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-9.8]   Dedup: 4.6->[7.7-9.8]
I20240426 13:20:07.859786 25100 pending_rounds.cc:76] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 6
I20240426 13:20:07.800618 27759 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:07.810966 27923 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:07.861678 24995 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.948s (SENT)
I20240426 13:20:07.872606 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:07.862474 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2015) took 1070 ms. Trace:
W20240426 13:20:07.858116 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.004s	sys 0.018s
I20240426 13:20:07.859575 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.811798 28034 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.860256 25100 pending_rounds.cc:97] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240426 13:20:07.801355 27916 raft_consensus.cc:2741] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20240426 13:20:07.875874 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.005s	sys 0.018s
W20240426 13:20:07.854463 27630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.790s	user 0.029s	sys 0.024s
I20240426 13:20:07.875582 24947 rpcz_store.cc:276] 0426 13:20:06.791958 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.330685 (+538727us) service_pool.cc:225] Handling call
0426 13:20:07.330974 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.745262 (+414288us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.862213 (+116951us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:07.862410 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":948391296}
I20240426 13:20:07.876581 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.887492 24951 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.879860 25100 raft_consensus.cc:2866] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.786114 27869 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.857439 27498 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.009s	sys 0.013s
I20240426 13:20:07.882656 26679 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:07.883320 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.854787 27653 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:07.895197 26679 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:07.872767 27810 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.003s	sys 0.020s
I20240426 13:20:07.858228 27633 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.860487 27853 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.890219 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.860675 27438 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:07.870545 28036 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:07.918910 28036 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.919476 28036 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:07.892616 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.855110 27451 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.898017 27900 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.824s	user 0.008s	sys 0.001s
I20240426 13:20:07.898517 27943 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.903035 27647 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.903723 27755 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:07.889292 27923 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:07.906210 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2101) took 1609 ms. Trace:
W20240426 13:20:07.942174 25022 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.011s	sys 0.014s
W20240426 13:20:07.942803 25022 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.011s	sys 0.014s
W20240426 13:20:07.944824 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61593.2us
I20240426 13:20:07.888044 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.917084 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.946516 25099 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.886587 27630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.029s	sys 0.025s
I20240426 13:20:07.870523 27759 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:07.920935 25101 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.007s	sys 0.017s
W20240426 13:20:07.895814 27498 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.010s	sys 0.013s
I20240426 13:20:07.923568 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.956377 25588 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.007s	sys 0.018s
W20240426 13:20:07.906119 27810 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.003s	sys 0.021s
I20240426 13:20:07.945844 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:07.947996 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.010s	sys 0.011s
W20240426 13:20:07.958058 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.010s	sys 0.012s
W20240426 13:20:07.947683 25026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.011s	sys 0.010s
I20240426 13:20:07.935978 25029 rpcz_store.cc:276] 0426 13:20:06.277647 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.277825 (+   178us) service_pool.cc:225] Handling call
0426 13:20:06.278092 (+   267us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:06.278263 (+   171us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.140307 (+862044us) log.cc:842] Serialized 14 byte log entry
0426 13:20:07.886494 (+746187us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:07.886582 (+    88us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:07.886637 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.886709 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:07.886817 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4701,"prepare.queue_time_us":1145741,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":16317,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":287,"thread_start_us":1563982,"threads_started":2}
I20240426 13:20:07.959403 26679 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.959800 26679 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.960130 26679 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.960156 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:07.960819 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.011s	sys 0.024s
W20240426 13:20:07.961145 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.011s	sys 0.024s
I20240426 13:20:07.961113 25029 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:07.955729 25101 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.007s	sys 0.017s
I20240426 13:20:07.900900 28065 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:07.858027 28062 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.888681 28047 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:07.922279 27853 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.957520 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69594.4us
I20240426 13:20:07.958792 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:07.963768 25026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.012s	sys 0.010s
I20240426 13:20:07.963716 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:07.965159 28049 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.004s	sys 0.020s
I20240426 13:20:07.930721 27900 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.957221 25588 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.008s	sys 0.018s
I20240426 13:20:07.971490 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:07.981045 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:07.987910 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 663000us
I20240426 13:20:07.988514 24954 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9.
I20240426 13:20:07.971534 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:07.981640 25103 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.005s	sys 0.025s
I20240426 13:20:07.981030 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:07.957738 27730 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.031s	sys 0.023s
I20240426 13:20:07.983522 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.994292 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.956655 27759 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:07.979439 27725 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.001s	sys 0.026s
W20240426 13:20:07.991094 25103 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.005s	sys 0.025s
I20240426 13:20:08.002594 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.003032 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.003324 25624 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:08.004170 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.003677 25024 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:08.004761 25029 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.005633 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.006531 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.007547 24956 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:07.993223 24995 connection.cc:232] RPC call timeout handler was delayed by -1.0057191470000362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
W20240426 13:20:07.983667 25025 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.001s	sys 0.026s
I20240426 13:20:07.972330 27943 raft_consensus.cc:2741] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20240426 13:20:07.984452 28049 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.005s	sys 0.021s
I20240426 13:20:07.989742 27653 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:20:08.010089 25025 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.001s	sys 0.026s
W20240426 13:20:08.010473 24995 connection.cc:232] RPC call timeout handler was delayed by -1.004513162999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20240426 13:20:08.025738 24995 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.988s (SENT)
I20240426 13:20:08.026350 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:08.011443 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.006s	sys 0.023s
I20240426 13:20:08.016803 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.994226 27730 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.032s	sys 0.024s
W20240426 13:20:08.021111 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.034s
W20240426 13:20:08.006616 27725 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.002s	sys 0.027s
I20240426 13:20:08.032407 27725 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:08.032446 28047 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:08.032951 27725 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:08.033226 27799 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.009s	sys 0.020s
I20240426 13:20:08.033448 27725 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:08.033584 27799 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.009s	sys 0.020s
I20240426 13:20:08.034121 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1995) took 1726 ms. Trace:
I20240426 13:20:08.034513 24952 rpcz_store.cc:276] 0426 13:20:06.307506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.703077 (+395571us) service_pool.cc:225] Handling call
0426 13:20:06.719621 (+ 16544us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:06.755714 (+ 36093us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:06.755762 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:06.810987 (+ 55225us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.737232 (+926245us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:07.737286 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.737373 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.033904 (+296531us) raft_consensus.cc:1671] finished
0426 13:20:08.033924 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.034065 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2821,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":16631,"thread_start_us":901751,"threads_started":1,"wal-append.queue_time_us":1178090}
I20240426 13:20:07.968310 27911 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.036206 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.036935 24952 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.037696 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.037994 27911 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.038236 24952 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:08.038686 27911 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:20:08.039273 27911 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.024819 27923 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.000304 28065 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:08.028669 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.041201 25024 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:08.030356 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.006s	sys 0.024s
W20240426 13:20:08.030565 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.000s	sys 0.034s
I20240426 13:20:08.003628 27916 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:08.048899 26679 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.004925 27382 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.086551 26679 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.089031 27382 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.026993 28061 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.069s
I20240426 13:20:08.090970 28061 consensus_queue.cc:1225] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:20:08.091759 27382 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.096320 24955 raft_consensus.cc:2396] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:08.097610 27759 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.051506 27916 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.101737 27847 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:08.103464 27911 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.106936 24947 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.026801 27653 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.100610 27916 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.981146 28062 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:08.133158 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.002s	user 0.022s	sys 0.025s
W20240426 13:20:08.133605 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.022s	sys 0.025s
I20240426 13:20:08.122740 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.045435 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
W20240426 13:20:08.105947 27271 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.000s
I20240426 13:20:08.135215 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.099792 27759 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.128177 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:08.109642 27847 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:08.122967 27677 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.059s	user 0.011s	sys 0.003s
W20240426 13:20:08.133381 27350 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.184s	user 0.007s	sys 0.006s
I20240426 13:20:08.144060 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.122766 27653 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:08.114660 27708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.020s	sys 0.013s
W20240426 13:20:08.121307 27928 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.012s	sys 0.016s
I20240426 13:20:08.148816 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.144800 24954 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:08.154330 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 539) took 1323 ms. Trace:
W20240426 13:20:08.121474 27868 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.008s	sys 0.019s
W20240426 13:20:08.133949 27707 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.005s	sys 0.024s
W20240426 13:20:08.144806 25104 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.018s	sys 0.029s
W20240426 13:20:08.185890 25104 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.018s	sys 0.034s
I20240426 13:20:08.185971 24956 rpcz_store.cc:276] 0426 13:20:06.830497 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.364730 (+534233us) service_pool.cc:225] Handling call
0426 13:20:07.400122 (+ 35392us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.008062 (+607940us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.154055 (+145993us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540038efd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.154260 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396514176}
I20240426 13:20:08.187283 25024 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:20:08.192857 25615 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:08.135332 27869 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.198038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.198844 24955 raft_consensus.cc:2362] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8. Re-sending same reply.
I20240426 13:20:08.190577 25028 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:20:08.209765 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.035s (SENT)
I20240426 13:20:08.229823 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 542) took 1343 ms. Trace:
W20240426 13:20:08.232920 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.191s (SENT)
I20240426 13:20:08.233443 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:08.232975 24952 rpcz_store.cc:276] 0426 13:20:06.886539 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.156637 (+1270098us) service_pool.cc:225] Handling call
0426 13:20:08.191986 (+ 35349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.229387 (+ 37401us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.229427 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.229495 (+    68us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.229537 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.229617 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.229757 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.187400 27707 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.936s	user 0.009s	sys 0.024s
I20240426 13:20:08.237831 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:08.269209 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.309s (SENT)
I20240426 13:20:08.235705 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.274015 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:08.269327 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 544) took 1341 ms. Trace:
I20240426 13:20:08.276290 24956 rpcz_store.cc:276] 0426 13:20:06.927551 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.187262 (+1259711us) service_pool.cc:225] Handling call
0426 13:20:08.239138 (+ 51876us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.268999 (+ 29861us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.269023 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.269076 (+    53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.269104 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.269154 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.269273 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.269881 24948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.028s	sys 0.026s
W20240426 13:20:08.277711 24948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.028s	sys 0.026s
W20240426 13:20:08.230545 28025 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.963s	user 0.015s	sys 0.000s
W20240426 13:20:08.228551 27708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.024s	sys 0.013s
I20240426 13:20:08.147320 28047 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:08.296017 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1996) took 1988 ms. Trace:
I20240426 13:20:08.281723 24955 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:08.284444 24952 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 4. Ops: []
I20240426 13:20:08.341935 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 546) took 1380 ms. Trace:
W20240426 13:20:08.185894 27868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.010s	sys 0.021s
W20240426 13:20:08.228750 27928 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.012s	sys 0.019s
I20240426 13:20:08.145316 27271 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.290823 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.801s (SENT)
I20240426 13:20:08.327695 27900 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.202836 27451 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.321980 24951 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:08.328342 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:08.329587 24956 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:08.282716 27943 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:08.281195 27677 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:08.160288 27350 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.378500 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 550) took 1297 ms. Trace:
I20240426 13:20:08.383822 24956 rpcz_store.cc:276] 0426 13:20:07.080927 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.277525 (+1196598us) service_pool.cc:225] Handling call
0426 13:20:08.320927 (+ 43402us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.378174 (+ 57247us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.378426 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.320981 25621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.013s	sys 0.031s
I20240426 13:20:08.307255 28047 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.342409 24952 rpcz_store.cc:276] 0426 13:20:06.961123 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.234640 (+1273517us) service_pool.cc:225] Handling call
0426 13:20:08.284253 (+ 49613us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.341610 (+ 57357us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.341852 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.343501 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 442331us
I20240426 13:20:08.389384 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 541) took 1541 ms. Trace:
W20240426 13:20:08.344249 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 744542us
W20240426 13:20:08.355008 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 349418us
I20240426 13:20:08.389807 24947 rpcz_store.cc:276] 0426 13:20:06.847579 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.155745 (+1308166us) service_pool.cc:225] Handling call
0426 13:20:08.199551 (+ 43806us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.236987 (+ 37436us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.272256 (+ 35269us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.272578 (+   322us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:08.272749 (+   171us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.273035 (+   286us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.389225 (+116190us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b54002cb0d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.389330 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99970944}
I20240426 13:20:08.391567 24947 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:08.355990 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 545) took 1413 ms. Trace:
I20240426 13:20:08.395787 24955 rpcz_store.cc:276] 0426 13:20:06.942446 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.206118 (+1263672us) service_pool.cc:225] Handling call
0426 13:20:08.281534 (+ 75416us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.355704 (+ 74170us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.355931 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.340879 27708 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:08.400697 27708 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.404376 27708 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.414676 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 552) took 1333 ms. Trace:
I20240426 13:20:08.415414 24947 rpcz_store.cc:276] 0426 13:20:07.081485 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.391009 (+1309524us) service_pool.cc:225] Handling call
0426 13:20:08.391352 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.414341 (+ 22989us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.414610 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.361438 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1804) took 1249 ms. Trace:
I20240426 13:20:08.268515 25024 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:08.269421 28065 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.103s
I20240426 13:20:08.379885 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 551) took 1298 ms. Trace:
I20240426 13:20:08.341990 24948 rpcz_store.cc:276] 0426 13:20:06.307795 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.705217 (+397422us) service_pool.cc:225] Handling call
0426 13:20:06.705617 (+   400us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:06.705825 (+   208us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.515340 (+809515us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.295550 (+780210us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.295626 (+    76us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.295688 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.295789 (+   101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.295949 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":963587,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":19194,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9101,"thread_start_us":1557411,"threads_started":2}
I20240426 13:20:08.380743 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.424657 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.425364 24954 rpcz_store.cc:276] 0426 13:20:07.081215 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.280572 (+1199357us) service_pool.cc:225] Handling call
0426 13:20:08.327059 (+ 46487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.379554 (+ 52495us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.379821 (+   267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.428530 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1648) took 1346 ms. Trace:
I20240426 13:20:08.429056 24947 rpcz_store.cc:276] 0426 13:20:07.082031 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.416599 (+1334568us) service_pool.cc:225] Handling call
0426 13:20:08.416974 (+   375us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.428104 (+ 11130us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.428143 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.428199 (+    56us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.428242 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.428322 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.428460 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.421496 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.413051 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:08.424001 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144434us
W20240426 13:20:08.432838 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.959s (SENT)
I20240426 13:20:08.434062 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:08.424101 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.436414 24952 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.424916 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.355626 27410 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.438519 24953 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:08.437407 24950 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:08.399164 25621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.013s	sys 0.032s
I20240426 13:20:08.387383 28047 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:08.442788 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.447594 24948 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:08.467981 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.468276 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1656) took 1372 ms. Trace:
I20240426 13:20:08.468626 24948 rpcz_store.cc:276] 0426 13:20:07.095776 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.425583 (+1329807us) service_pool.cc:225] Handling call
0426 13:20:08.425897 (+   314us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.467992 (+ 42095us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.468224 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.382699 27853 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.471253 24921 leader_election.cc:398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.471578 24921 leader_election.cc:401] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240426 13:20:08.442365 24947 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:20:08.409868 27708 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.451714 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:08.480990 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.017s	sys 0.007s
W20240426 13:20:08.481460 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.017s	sys 0.007s
W20240426 13:20:08.476480 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 469952us
I20240426 13:20:08.477356 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.288972 27847 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.477576 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1660) took 1215 ms. Trace:
I20240426 13:20:08.424633 27932 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:08.420182 27677 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.470778 24948 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:20:08.476750 24952 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 13:20:08.506124 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1662) took 1202 ms. Trace:
I20240426 13:20:08.506603 24952 rpcz_store.cc:276] 0426 13:20:07.305326 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.441966 (+1136640us) service_pool.cc:225] Handling call
0426 13:20:08.476357 (+ 34391us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.505774 (+ 29417us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.506057 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.492604 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102359us
I20240426 13:20:08.487403 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:08.508781 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.418877 25101 rpcz_store.cc:276] 0426 13:20:07.111831 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.212178 (+100347us) service_pool.cc:225] Handling call
0426 13:20:07.212456 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.212680 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.966572 (+753892us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.361001 (+394429us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.361076 (+    75us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.361144 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.361231 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.361372 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":666908,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":622,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":13982,"thread_start_us":1143377,"threads_started":2}
I20240426 13:20:08.492551 24947 rpcz_store.cc:276] 0426 13:20:07.262279 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.440253 (+1177974us) service_pool.cc:225] Handling call
0426 13:20:08.441038 (+   785us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.475326 (+ 34288us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.477276 (+  1950us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.503454 25099 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.001s
I20240426 13:20:08.495278 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.510453 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:08.505828 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2016) took 1107 ms. Trace:
I20240426 13:20:08.513671 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.514089 24948 rpcz_store.cc:276] 0426 13:20:07.398180 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.469350 (+1071170us) service_pool.cc:225] Handling call
0426 13:20:08.470420 (+  1070us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.505564 (+ 35144us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.505761 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1663104}
I20240426 13:20:08.513666 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1544) took 1110 ms. Trace:
I20240426 13:20:08.515377 25024 rpcz_store.cc:276] 0426 13:20:07.403400 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.495010 (+1091610us) service_pool.cc:225] Handling call
0426 13:20:08.513605 (+ 18595us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.517657 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.470921 27869 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:08.518754 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:08.520399 24947 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:08.523553 24948 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20240426 13:20:08.523932 24948 pending_rounds.cc:76] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:08.524291 24948 pending_rounds.cc:97] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20240426 13:20:08.524518 24948 raft_consensus.cc:2866] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:08.525452 26679 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:08.526464 26679 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:08.526947 26679 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.527328 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.528498 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.529553 25024 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:08.466331 27410 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.527878 26679 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.531618 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.536199 24947 raft_consensus.cc:2362] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8. Re-sending same reply.
I20240426 13:20:08.535072 24956 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:08.536846 26679 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:08.506733 27900 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.537238 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1670) took 1009 ms. Trace:
I20240426 13:20:08.545094 24956 rpcz_store.cc:276] 0426 13:20:07.528118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.517018 (+988900us) service_pool.cc:225] Handling call
0426 13:20:08.534752 (+ 17734us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.536988 (+  2236us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.537190 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.545094 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1668) took 1112 ms. Trace:
I20240426 13:20:08.546334 24951 rpcz_store.cc:276] 0426 13:20:07.432442 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.894209 (+461767us) service_pool.cc:225] Handling call
0426 13:20:08.544895 (+650686us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000794d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.545038 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838136064}
I20240426 13:20:08.522421 27869 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.542347 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.543556 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1801) took 1432 ms. Trace:
I20240426 13:20:08.517038 27498 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:08.517361 27847 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:08.522131 27928 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:08.522287 28097 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:08.541982 25029 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.519s	user 0.007s	sys 0.000s
I20240426 13:20:08.550772 25098 raft_consensus.cc:1203] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 6.4->[7.5-7.5]   Dedup: 7.5->[]
I20240426 13:20:08.563195 25029 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:08.551410 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1807) took 1431 ms. Trace:
I20240426 13:20:08.521629 28062 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.516280 27932 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:08.568488 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47782.9us
I20240426 13:20:08.565306 25098 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 9. Ops: [7.5-7.5]
W20240426 13:20:08.554855 27946 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.095s	user 0.010s	sys 0.002s
I20240426 13:20:08.552762 25103 rpcz_store.cc:276] 0426 13:20:07.111122 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.210798 (+ 99676us) service_pool.cc:225] Handling call
0426 13:20:07.211108 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.211290 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.002495 (+791205us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.543195 (+540700us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.543267 (+    72us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.543321 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.543382 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.543491 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49682,"prepare.queue_time_us":638961,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":69956,"thread_start_us":1259582,"threads_started":2}
I20240426 13:20:08.569453 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.520181 27677 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.549708 28035 raft_consensus.cc:2741] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:08.604671 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.605173 27759 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:08.605772 27759 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:08.605986 28074 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20240426 13:20:08.606158 27759 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.606750 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.554929 27869 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:08.608120 28082 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20240426 13:20:08.608345 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.608704 24956 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20240426 13:20:08.609028 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:08.609776 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.609944 27900 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:08.610306 27900 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.610689 27900 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:08.610893 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.612214 27647 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:08.609779 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.612705 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2103) took 1411 ms. Trace:
I20240426 13:20:08.613013 27647 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:08.613073 25030 rpcz_store.cc:276] 0426 13:20:07.201761 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.201879 (+   118us) service_pool.cc:225] Handling call
0426 13:20:07.267797 (+ 65918us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:07.288646 (+ 20849us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.288675 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:07.295989 (+  7314us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.043323 (+747334us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.043365 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.043433 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.612496 (+569063us) raft_consensus.cc:1671] finished
0426 13:20:08.612514 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.612652 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22,"raft.queue_time_us":20659,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":7104,"thread_start_us":722335,"threads_started":1,"wal-append.queue_time_us":1166744}
I20240426 13:20:08.614053 27647 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.615609 27738 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:08.594489 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.557365 27498 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.646382 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:08.603366 28087 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20240426 13:20:08.603499 25103 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:08.613111 24947 raft_consensus.cc:2362] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
W20240426 13:20:08.616377 28080 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20240426 13:20:08.604734 27677 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.564497 27928 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:08.562813 28088 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20240426 13:20:08.617332 27271 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.562809 27451 raft_consensus.cc:2741] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:08.629957 27946 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.612337 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.570587 28097 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.644233 28085 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20240426 13:20:08.644081 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1534) took 1606 ms. Trace:
I20240426 13:20:08.675109 27932 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.645320 25028 raft_consensus.cc:2396] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:08.705185 27498 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:08.595947 28062 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:20:08.645870 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.779953 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:08.780411 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135272us
I20240426 13:20:08.781248 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:08.748704 27738 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:08.605104 28093 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20240426 13:20:08.611907 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:08.645349 27725 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.611s	user 0.006s	sys 0.004s
I20240426 13:20:08.671590 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.694924 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 607) took 1251 ms. Trace:
I20240426 13:20:08.695614 27900 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.694196 27869 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.675613 28025 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.675240 27759 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.698240 27410 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.816419 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 553) took 1013 ms. Trace:
I20240426 13:20:08.816659 27869 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.802939 27451 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:08.817338 28025 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.817756 27451 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.818794 24921 leader_election.cc:398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.819371 28025 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.819653 27451 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:08.731750 24995 leader_election.cc:334] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 3.000s (SENT)
I20240426 13:20:08.779939 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:08.798842 27911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.006s	sys 0.004s
I20240426 13:20:08.598088 28055 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:08.832208 28055 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.833556 28097 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.834275 28097 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:08.834705 28097 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.834858 28055 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.835237 28097 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:08.751955 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.826471 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1541) took 1773 ms. Trace:
I20240426 13:20:08.836030 25622 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:08.837626 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1809) took 1444 ms. Trace:
I20240426 13:20:08.836792 25026 rpcz_store.cc:276] 0426 13:20:07.053534 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.137779 (+ 84245us) service_pool.cc:225] Handling call
0426 13:20:07.159409 (+ 21630us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.159622 (+   213us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.159645 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.198420 (+ 38775us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.981448 (+783028us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:07.981486 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.981554 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.826295 (+844741us) raft_consensus.cc:1671] finished
0426 13:20:08.826313 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.826413 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":675838,"threads_started":1,"wal-append.queue_time_us":1034774}
I20240426 13:20:08.838115 25096 rpcz_store.cc:276] 0426 13:20:07.392851 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.392979 (+   128us) service_pool.cc:225] Handling call
0426 13:20:07.413332 (+ 20353us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.414038 (+   706us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.414871 (+   833us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.420859 (+  5988us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.958534 (+537675us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:07.964305 (+  5771us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.964410 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.837354 (+872944us) raft_consensus.cc:1671] finished
0426 13:20:08.837382 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.837556 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":534440,"threads_started":1,"wal-append.queue_time_us":909954}
W20240426 13:20:08.677423 27916 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.535s	user 0.004s	sys 0.004s
I20240426 13:20:08.756110 24948 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.843333 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:08.674312 28034 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.785s	user 0.009s	sys 0.000s
I20240426 13:20:08.844462 28034 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.677989 28036 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.757s	user 0.009s	sys 0.000s
I20240426 13:20:08.769335 25023 rpcz_store.cc:276] 0426 13:20:07.037677 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.128961 (+ 91284us) service_pool.cc:225] Handling call
0426 13:20:07.129245 (+   284us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.130669 (+  1424us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.130697 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.131194 (+   497us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.140335 (+1009141us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:08.141114 (+   779us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.150355 (+  9241us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.150432 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.616484 (+466052us) raft_consensus.cc:1671] finished
0426 13:20:08.616503 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.644005 (+ 27502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8966,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":10028,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6508,"thread_start_us":1002497,"threads_started":1,"wal-append.queue_time_us":1293510}
W20240426 13:20:08.644692 28076 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
I20240426 13:20:08.647264 24951 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:08.850721 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 294479us
I20240426 13:20:08.860472 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:08.865705 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.753383 26679 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:08.933547 28106 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20240426 13:20:08.933441 26679 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.934362 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.935559 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.935922 26679 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.936435 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.777516 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1524) took 1349 ms. Trace:
I20240426 13:20:08.570272 28061 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.071s
I20240426 13:20:08.752765 27928 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:08.754278 27868 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:08.937161 28061 consensus_queue.cc:1225] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:08.937506 27868 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.938437 27868 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.936928 25098 rpcz_store.cc:276] 0426 13:20:07.427768 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.442013 (+ 14245us) service_pool.cc:225] Handling call
0426 13:20:08.550665 (+1108652us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.569185 (+ 18520us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.777316 (+208131us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540009d8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.777459 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2548854656}
I20240426 13:20:08.943609 25098 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:08.778142 25618 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.703907 28047 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.802439 27725 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.944885 27928 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.945513 27725 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.945701 25029 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:20:08.809989 28092 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20240426 13:20:08.798049 25621 rpcz_store.cc:276] 0426 13:20:07.443955 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.449746 (+  5791us) service_pool.cc:225] Handling call
0426 13:20:07.496532 (+ 46786us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.550794 (+ 54262us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.556868 (+  6074us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.663402 (+106534us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.479157 (+815755us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:08.491799 (+ 12642us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.505584 (+ 13785us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.694644 (+189060us) raft_consensus.cc:1671] finished
0426 13:20:08.694681 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.694855 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4265,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13997,"thread_start_us":672875,"threads_started":1,"wal-append.queue_time_us":961555}
I20240426 13:20:08.812314 28096 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.947113 27928 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.947043 28096 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.947662 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2115) took 1727 ms. Trace:
I20240426 13:20:08.813140 27759 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.814697 27900 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.820421 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2111) took 1607 ms. Trace:
I20240426 13:20:08.699030 27350 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.953923 25031 rpcz_store.cc:276] 0426 13:20:07.213366 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.307744 (+ 94378us) service_pool.cc:225] Handling call
0426 13:20:07.409840 (+102096us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.415672 (+  5832us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.415714 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.489427 (+ 73713us) log.cc:842] Serialized 548 byte log entry
0426 13:20:08.043788 (+554361us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.043829 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.043895 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.820106 (+776211us) raft_consensus.cc:1671] finished
0426 13:20:08.820134 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.820346 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":553964,"threads_started":1,"wal-append.queue_time_us":1323511}
W20240426 13:20:08.820693 28101 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20240426 13:20:08.821867 24956 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.824532 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:08.740514 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.957923 24947 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:08.784188 27923 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.978600 27923 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.979192 27498 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.742115 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147586us
I20240426 13:20:08.830003 27738 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.008929 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1812) took 1104 ms. Trace:
I20240426 13:20:08.819662 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:08.772282 27799 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20240426 13:20:08.847746 24948 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.866639 25096 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:08.868319 27451 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:08.921660 28034 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.922401 27946 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.015708 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.922905 25103 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:09.016026 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 284300us
I20240426 13:20:08.923024 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2108) took 1718 ms. Trace:
W20240426 13:20:09.017161 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 3.322s (SENT)
W20240426 13:20:08.923800 28072 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20240426 13:20:08.859120 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.023677 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.024369 25619 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.025167 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.859249 28036 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:09.027133 28036 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.027707 25097 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:09.028347 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1647) took 1946 ms. Trace:
I20240426 13:20:09.028807 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.016978 25022 rpcz_store.cc:276] 0426 13:20:07.204149 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.252128 (+ 47979us) service_pool.cc:225] Handling call
0426 13:20:07.273648 (+ 21520us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:07.306258 (+ 32610us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:20:07.943347 (+637089us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.353849 (+410502us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:08.354497 (+   648us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.354673 (+   176us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:08.354717 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.354803 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.922793 (+567990us) raft_consensus.cc:1671] finished
0426 13:20:08.922816 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.922969 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18770,"prepare.queue_time_us":350229,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":6433,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":4962,"thread_start_us":980447,"threads_started":2,"wal-append.queue_time_us":816229}
I20240426 13:20:09.029335 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1659) took 1834 ms. Trace:
I20240426 13:20:09.017093 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:08.850358 27911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.006s	sys 0.005s
I20240426 13:20:08.931157 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.932268 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:08.911036 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48152us
I20240426 13:20:08.934453 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.948011 25029 rpcz_store.cc:276] 0426 13:20:07.219775 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.563987 (+1344212us) service_pool.cc:225] Handling call
0426 13:20:08.564306 (+   319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.946114 (+381808us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.947435 (+  1321us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400242f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.947611 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876686208}
I20240426 13:20:09.033718 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.034310 25029 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:08.867576 27916 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.699041 27653 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.502s	user 0.006s	sys 0.000s
I20240426 13:20:08.945513 28047 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:08.955147 28107 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20240426 13:20:08.955787 28055 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
W20240426 13:20:08.963940 27964 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.163s	user 0.008s	sys 0.007s
I20240426 13:20:08.968315 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.968667 25621 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:08.968924 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1665) took 1587 ms. Trace:
I20240426 13:20:08.970642 28096 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.564378 25104 rpcz_store.cc:276] 0426 13:20:07.119506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.257270 (+137764us) service_pool.cc:225] Handling call
0426 13:20:07.290331 (+ 33061us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.290562 (+   231us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.186424 (+895862us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.550386 (+363962us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.550440 (+    54us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.550486 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.550554 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.551353 (+   799us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36809,"prepare.queue_time_us":618122,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":36971,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3290,"thread_start_us":1202827,"threads_started":2}
I20240426 13:20:08.971565 27410 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240426 13:20:08.974606 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 547) took 1938 ms (client timeout 1752 ms). Trace:
I20240426 13:20:08.974745 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:08.975834 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.976013 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2105) took 1773 ms. Trace:
W20240426 13:20:08.977911 28086 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.506s	user 0.001s	sys 0.001s
I20240426 13:20:08.817001 25100 rpcz_store.cc:276] 0426 13:20:07.803092 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.810662 (+  7570us) service_pool.cc:225] Handling call
0426 13:20:07.811316 (+   654us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:07.937025 (+125709us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:07.937060 (+    35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:07.937596 (+   536us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.474629 (+537033us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.474711 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.474815 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.816109 (+341294us) raft_consensus.cc:1671] finished
0426 13:20:08.816143 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.816349 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":465506,"threads_started":1,"wal-append.queue_time_us":862723}
I20240426 13:20:09.031023 27647 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:08.840224 28025 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
W20240426 13:20:08.981199 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 471100us
I20240426 13:20:09.088948 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1810) took 1657 ms. Trace:
I20240426 13:20:08.994880 25101 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:09.013868 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:09.090543 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76757us
I20240426 13:20:09.009361 25105 rpcz_store.cc:276] 0426 13:20:07.904270 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.906652 (+  2382us) service_pool.cc:225] Handling call
0426 13:20:07.921920 (+ 15268us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.156749 (+234829us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.156777 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.225429 (+ 68652us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.500556 (+275127us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:08.500591 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.994082 (+493491us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.994136 (+    54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.994179 (+    43us) raft_consensus.cc:1671] finished
0426 13:20:08.994192 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.008859 (+ 14667us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":428,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":21785,"spinlock_wait_cycles":539274624,"thread_start_us":263585,"threads_started":1,"wal-append.queue_time_us":631649}
I20240426 13:20:08.923632 27869 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:09.022425 27350 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.022857 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.023048 28049 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.028810 24955 rpcz_store.cc:276] 0426 13:20:07.081764 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.396708 (+1314944us) service_pool.cc:225] Handling call
0426 13:20:08.446747 (+ 50039us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.473828 (+ 27081us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.473863 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.474306 (+   443us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.705804 (+231498us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.706177 (+   373us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.706266 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.028146 (+321880us) raft_consensus.cc:1671] finished
0426 13:20:09.028166 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.028294 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4550,"raft.queue_time_us":3685,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":12840,"thread_start_us":174349,"threads_started":1,"wal-append.queue_time_us":516380}
I20240426 13:20:09.105388 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.031463 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:09.045892 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 226297us
I20240426 13:20:09.107146 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.046710 24947 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:08.973235 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:09.048043 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.048786 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.108683 24956 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.051091 26679 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:09.050786 24953 rpcz_store.cc:276] 0426 13:20:07.381891 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.659328 (+277437us) service_pool.cc:225] Handling call
0426 13:20:08.968662 (+1309334us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54002cbfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.968830 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1556543872}
I20240426 13:20:09.051910 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.052251 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.016441 27451 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:09.058110 24950 rpcz_store.cc:269] 0426 13:20:07.036069 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.117384 (+ 81315us) service_pool.cc:225] Handling call
0426 13:20:08.974351 (+1856967us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.974499 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2977547520}
I20240426 13:20:09.105865 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.116714 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.118031 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.118295 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.118700 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:09.119457 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.058566 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.065244 25029 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:08.980765 27923 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:09.070497 28075 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.782s	user 0.003s	sys 0.001s
I20240426 13:20:09.057744 25025 rpcz_store.cc:276] 0426 13:20:07.202327 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.202444 (+   117us) service_pool.cc:225] Handling call
0426 13:20:07.202699 (+   255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.202891 (+   192us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.023819 (+820928us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.443584 (+419765us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.822527 (+378943us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351
0426 13:20:08.830079 (+  7552us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:08.975549 (+145470us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:20:08.975687 (+   138us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.975735 (+    48us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.975778 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.975851 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.975952 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":527827,"prepare.queue_time_us":408596,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":558381,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":12645,"spinlock_wait_cycles":109254400,"thread_start_us":1130850,"threads_started":2,"wal-append.queue_time_us":542067}
I20240426 13:20:09.053064 27808 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240426 13:20:09.053170 27853 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.053313 27847 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20240426 13:20:09.059558 27911 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20240426 13:20:09.084499 27833 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:09.053539 28036 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20240426 13:20:09.086171 28094 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.620s	user 0.001s	sys 0.001s
I20240426 13:20:09.056933 28047 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.030s
I20240426 13:20:09.087126 28062 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:09.173575 28062 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:09.174039 28062 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.046343 27271 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.142397 27810 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:09.089854 25102 rpcz_store.cc:276] 0426 13:20:07.431365 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.450079 (+ 18714us) service_pool.cc:225] Handling call
0426 13:20:07.493172 (+ 43093us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.496531 (+  3359us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:08.011173 (+514642us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.495143 (+483970us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.509839 (+ 14696us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.509985 (+   146us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:08.510016 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.510077 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.088754 (+578677us) raft_consensus.cc:1671] finished
0426 13:20:09.088776 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.088897 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19349,"prepare.queue_time_us":412460,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":31972,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":9772,"thread_start_us":969961,"threads_started":2,"wal-append.queue_time_us":731740}
I20240426 13:20:09.091630 25619 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:09.092159 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1666) took 1693 ms. Trace:
I20240426 13:20:09.110201 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.139586 28131 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.115134 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2113) took 1901 ms. Trace:
I20240426 13:20:09.120899 24956 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.029757 24954 rpcz_store.cc:276] 0426 13:20:07.194952 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.427993 (+1233041us) service_pool.cc:225] Handling call
0426 13:20:08.470123 (+ 42130us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.504125 (+ 34002us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:08.662896 (+158771us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.008669 (+345773us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.029021 (+ 20352us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.029070 (+    49us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.029118 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.029191 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.029288 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20937,"prepare.queue_time_us":330279,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":47917,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":366940,"threads_started":2}
W20240426 13:20:09.122795 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 70611.3us
I20240426 13:20:09.187944 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.190251 24956 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:09.190619 24956 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:09.124353 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.193559 25619 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:20:09.130429 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 540) took 2294 ms. Trace:
I20240426 13:20:09.196195 24949 rpcz_store.cc:276] 0426 13:20:06.836118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.365400 (+529282us) service_pool.cc:225] Handling call
0426 13:20:08.974496 (+1609096us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.057896 (+ 83400us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.130255 (+ 72359us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54000785d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.130374 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3590358912}
I20240426 13:20:09.197646 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.198428 24949 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.199388 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.133270 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.130093 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.121713 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:09.129671 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.131733 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.201414 25029 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:09.126551 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.135445 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.139783 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.141568 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:09.143142 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.187319 25024 rpcz_store.cc:276] 0426 13:20:07.213868 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.560353 (+1346485us) service_pool.cc:225] Handling call
0426 13:20:08.560663 (+   310us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:08.982384 (+421721us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.982420 (+    36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:08.982950 (+   530us) log.cc:842] Serialized 52 byte log entry
0426 13:20:09.087277 (+104327us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b18000b1fa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:09.112597 (+ 25320us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:20:09.112649 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.112746 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.114807 (+  2061us) raft_consensus.cc:1671] finished
0426 13:20:09.114824 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.115067 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":416418,"raft.queue_time_us":417251,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":17132,"spinlock_wait_cycles":11081984,"thread_start_us":25069,"threads_started":1,"wal-append.queue_time_us":116103}
I20240426 13:20:09.206529 24950 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:09.207271 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.144162 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 557) took 1022 ms. Trace:
I20240426 13:20:09.207731 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.144536 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1815) took 1239 ms. Trace:
I20240426 13:20:09.047264 27653 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.208189 25101 rpcz_store.cc:276] 0426 13:20:08.121567 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.554016 (+432449us) service_pool.cc:225] Handling call
0426 13:20:08.994698 (+440682us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.089999 (+ 95301us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.143872 (+ 53873us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54000ea950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.144091 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012761472}
I20240426 13:20:09.210789 27653 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.105420 28055 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.123191 27451 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:09.150631 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.103464 28065 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.211619 28055 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.212342 28065 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.214219 28055 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:09.103289 28025 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:09.215437 28025 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.215956 25099 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.205750 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.095283 27637 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20240426 13:20:09.218412 27451 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.162717 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 405) took 1705 ms. Trace:
I20240426 13:20:09.162778 27498 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.219695 25100 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
W20240426 13:20:09.172597 28132 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:20:09.219841 25097 rpcz_store.cc:276] 0426 13:20:07.905321 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.029857 (+124536us) service_pool.cc:225] Handling call
0426 13:20:08.101460 (+ 71603us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.028093 (+926633us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.144241 (+116148us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:09.144479 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111356672}
I20240426 13:20:09.120024 27755 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20240426 13:20:09.220683 27451 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:09.112172 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 137904us
I20240426 13:20:09.184201 24952 rpcz_store.cc:276] 0426 13:20:07.400881 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.507576 (+1106695us) service_pool.cc:225] Handling call
0426 13:20:08.529476 (+ 21900us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.543975 (+ 14499us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.544006 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.545411 (+  1405us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.799176 (+253765us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.091442 (+292266us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.091770 (+   328us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.091861 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.091917 (+    56us) raft_consensus.cc:1671] finished
0426 13:20:09.091934 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.092096 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19064,"prepare.queue_time_us":30583,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":17118,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":32011,"thread_start_us":461427,"threads_started":2,"wal-append.queue_time_us":431200}
I20240426 13:20:09.223783 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.219578 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.224576 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:09.224964 24953 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.206414 24955 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:20:09.097632 27410 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:09.226635 27410 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.227064 27410 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.227138 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.224140 25024 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:09.191416 24956 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.228036 24955 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.208642 24950 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:09.232961 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.214320 28065 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:09.154146 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.235317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81281.4us
I20240426 13:20:09.132261 28104 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.228510 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.236624 28126 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.236048 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.237486 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.238101 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.238237 25022 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.238425 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.238813 25029 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.239288 27900 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.239874 27900 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:09.219635 25028 rpcz_store.cc:276] 0426 13:20:07.457704 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.967753 (+1510049us) service_pool.cc:225] Handling call
0426 13:20:08.971011 (+  3258us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.031318 (+ 60307us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.031349 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.038843 (+  7494us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.114325 (+ 75482us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:09.114664 (+   339us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.162413 (+ 47749us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.162485 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.162525 (+    40us) raft_consensus.cc:1671] finished
0426 13:20:09.162539 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.162661 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94692,"prepare.queue_time_us":329,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":88621,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":22164,"thread_start_us":58356,"threads_started":2,"wal-append.queue_time_us":41467}
I20240426 13:20:09.225788 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.121693 27350 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.156333 28096 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.247282 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.247974 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 404) took 1792 ms. Trace:
I20240426 13:20:09.246482 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.247231 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.244515 24947 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:09.229274 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.249735 24954 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20240426 13:20:09.250376 24954 pending_rounds.cc:76] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 4
I20240426 13:20:09.228829 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.251003 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 407) took 1768 ms. Trace:
I20240426 13:20:09.229758 24956 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.251505 25028 rpcz_store.cc:276] 0426 13:20:07.483093 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.242091 (+1758998us) service_pool.cc:225] Handling call
0426 13:20:09.250551 (+  8460us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.250720 (+   169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.250747 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.250787 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:09.250821 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.250869 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.250955 (+    86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.144264 27677 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:20:09.253949 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:09.249068 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 127431us
I20240426 13:20:09.257444 24955 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:09.257499 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:09.256973 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.243377 24952 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.229090 24949 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.259629 24954 pending_rounds.cc:97] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240426 13:20:09.260747 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.259007 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.264954 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.265506 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.265690 24948 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.268898 24955 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.273320 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.274860 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.277673 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.260725 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.285308 24954 raft_consensus.cc:2866] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.287699 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.288357 25024 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:09.249631 25031 rpcz_store.cc:276] 0426 13:20:07.455773 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.956812 (+1501039us) service_pool.cc:225] Handling call
0426 13:20:08.957229 (+   417us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.008808 (+ 51579us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.008851 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.009359 (+   508us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.114510 (+105151us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.114822 (+   312us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.247644 (+132822us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.247749 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.247790 (+    41us) raft_consensus.cc:1671] finished
0426 13:20:09.247805 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.247924 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77098,"prepare.queue_time_us":416,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":80212,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":25732,"thread_start_us":162101,"threads_started":2,"wal-append.queue_time_us":75906}
I20240426 13:20:09.297148 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2123) took 1630 ms. Trace:
I20240426 13:20:09.297619 25031 rpcz_store.cc:276] 0426 13:20:07.667090 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.296103 (+1629013us) service_pool.cc:225] Handling call
0426 13:20:09.296495 (+   392us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.296724 (+   229us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.296762 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.296820 (+    58us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.296865 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.296933 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.297082 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.299332 25031 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:09.143994 28124 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20240426 13:20:09.301182 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 409) took 1812 ms. Trace:
I20240426 13:20:09.145606 27853 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.308073 24947 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.307621 25028 rpcz_store.cc:276] 0426 13:20:07.488868 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.253521 (+1764653us) service_pool.cc:225] Handling call
0426 13:20:09.300752 (+ 47231us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.300909 (+   157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.300931 (+    22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.300967 (+    36us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.300996 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.301040 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.301136 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.303259 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.144917 28099 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.116s
I20240426 13:20:09.313994 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.312732 25022 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.310175 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:09.314523 24956 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:20:09.309947 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 410) took 1409 ms. Trace:
I20240426 13:20:09.320466 24956 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:09.320612 25028 rpcz_store.cc:276] 0426 13:20:07.900864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.309154 (+1408290us) service_pool.cc:225] Handling call
0426 13:20:09.309408 (+   254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.309555 (+   147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.309595 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.309648 (+    53us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.309697 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.309763 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.309872 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.322489 25028 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:09.247020 27810 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:09.316190 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.313436 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2126) took 1402 ms. Trace:
I20240426 13:20:09.323184 24952 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:09.323132 25031 rpcz_store.cc:276] 0426 13:20:07.897792 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.298737 (+1400945us) service_pool.cc:225] Handling call
0426 13:20:09.299114 (+   377us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.299789 (+   675us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.299984 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.323596 24955 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.325037 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.320930 24956 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:20:09.325301 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:09.325819 24956 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.327190 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.328008 24956 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.253657 28126 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.333420 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.334024 24954 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:09.261003 28061 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:09.341349 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.341949 24953 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.315747 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.332396 25022 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:09.166067 28120 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.149s
I20240426 13:20:09.261389 27451 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.329502 24955 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.241709 27271 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.342605 24949 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:09.284415 28104 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:09.314011 28086 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20240426 13:20:09.343454 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 121073us
I20240426 13:20:09.302946 27653 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:09.354387 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44313.5us
I20240426 13:20:09.356406 25618 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:09.314607 27633 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
W20240426 13:20:09.359565 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97170.2us
I20240426 13:20:09.360565 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.315701 28047 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.320144 27708 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:09.175083 28062 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:09.370056 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 228585us
W20240426 13:20:09.401167 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 274706us
I20240426 13:20:09.284704 27923 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.064s
I20240426 13:20:09.405478 24950 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:09.405898 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 954) took 1663 ms. Trace:
I20240426 13:20:09.406234 25022 rpcz_store.cc:276] 0426 13:20:07.742669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.358206 (+1615537us) service_pool.cc:225] Handling call
0426 13:20:09.405831 (+ 47625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.406968 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:09.403285 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.353379 24947 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.344930 27916 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.351248 27900 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:20:09.425791 28134 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:20:09.426548 24950 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 13:20:09.427281 28135 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:20:09.428551 27810 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.431385 27350 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.432114 28124 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:09.434935 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.435451 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.416728 24949 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:09.436796 24948 pending_rounds.cc:76] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:20:09.437673 24948 pending_rounds.cc:97] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:20:09.437974 24948 raft_consensus.cc:2866] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.419317 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.419494 26679 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:09.353271 28104 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.354575 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.418043 27755 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.003s
I20240426 13:20:09.420822 28061 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.417198 24953 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:09.419330 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94137.1us
I20240426 13:20:09.449787 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.364781 27725 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:09.428503 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2118) took 2074 ms. Trace:
I20240426 13:20:09.453219 25023 rpcz_store.cc:276] 0426 13:20:07.354211 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.900393 (+1546182us) service_pool.cc:225] Handling call
0426 13:20:08.919616 (+ 19223us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.997930 (+ 78314us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.997977 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.018413 (+ 20436us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.152273 (+133860us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.152314 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.152379 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.428337 (+275958us) raft_consensus.cc:1671] finished
0426 13:20:09.428353 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.428445 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77830,"raft.queue_time_us":77936,"spinlock_wait_cycles":6890240,"thread_start_us":93516,"threads_started":1,"wal-append.queue_time_us":212098}
I20240426 13:20:09.444492 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:09.419044 28100 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:20:09.410248 27633 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:09.446666 28097 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.611s	user 0.001s	sys 0.009s
I20240426 13:20:09.452155 24953 pending_rounds.cc:76] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:09.458492 24953 pending_rounds.cc:97] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:09.458806 24953 raft_consensus.cc:2866] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.459890 26679 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.460956 26679 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.462568 24954 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:09.458240 28126 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:09.465404 28061 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.466549 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2109) took 2262 ms. Trace:
I20240426 13:20:09.466890 25027 rpcz_store.cc:276] 0426 13:20:07.204406 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.296860 (+ 92454us) service_pool.cc:225] Handling call
0426 13:20:07.297187 (+   327us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:07.307586 (+ 10399us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.307619 (+    33us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:07.322884 (+ 15265us) log.cc:842] Serialized 78 byte log entry
0426 13:20:07.961777 (+638893us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:07.981729 (+ 19952us) log.cc:842] Serialized 14 byte log entry
0426 13:20:07.985103 (+  3374us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.985180 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.977316 (+992136us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:09.458739 (+481423us) raft_consensus.cc:1671] finished
0426 13:20:09.466334 (+  7595us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.466475 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23557,"prepare.queue_time_us":19772,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":8145,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":21896,"spinlock_wait_cycles":59144448,"thread_start_us":597186,"threads_started":1,"wal-append.queue_time_us":1004760}
I20240426 13:20:09.468421 25029 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 5.7->[7.8-9.9]   Dedup: 9.9->[]
I20240426 13:20:09.468533 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2117) took 2216 ms. Trace:
I20240426 13:20:09.438482 24952 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.475206 28097 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.477821 28097 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.447517 28062 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.488479 25029 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
W20240426 13:20:09.490801 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46413.2us
I20240426 13:20:09.492384 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:09.443195 28104 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:09.493652 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74399.1us
W20240426 13:20:09.537204 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.005s	sys 0.003s
W20240426 13:20:09.537766 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.005s	sys 0.003s
I20240426 13:20:09.468897 25026 rpcz_store.cc:276] 0426 13:20:07.251822 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.841405 (+1589583us) service_pool.cc:225] Handling call
0426 13:20:08.969132 (+127727us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.969331 (+   199us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:08.969633 (+   302us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.147368 (+177735us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.147438 (+    70us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:09.147510 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.147600 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.468385 (+320785us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54004169d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.468484 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70034,"prepare.queue_time_us":7240,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":38277,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":56766,"spinlock_wait_cycles":238727552,"thread_start_us":167560,"threads_started":1,"wal-append.queue_time_us":200751}
I20240426 13:20:09.451354 28025 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:20:09.540343 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66682.3us
I20240426 13:20:09.592891 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.612778 28035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.007s	sys 0.015s
I20240426 13:20:09.468711 27916 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:09.470371 27725 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.632519 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.614804 28035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.008s	sys 0.015s
I20240426 13:20:09.539597 27677 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:09.617981 28146 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20240426 13:20:09.541384 28034 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.595006 28131 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.619814 28119 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:09.642937 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.623142 27868 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 13:20:09.647603 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 288877us
I20240426 13:20:09.628693 27271 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:09.656554 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 164236us
W20240426 13:20:09.658011 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 208270us
I20240426 13:20:09.661624 27653 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.627964 27498 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:09.636770 27647 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:09.664754 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.677179 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.696481 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.663535 28119 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.638126 27410 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.639467 27916 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.707268 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.712117 25028 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:09.649179 28034 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.695387 27498 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.701143 27928 raft_consensus.cc:2741] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240426 13:20:09.731976 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.649853 28131 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.705399 27946 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:09.738065 25026 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:09.740531 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98553.5us
I20240426 13:20:09.740756 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.741344 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.742926 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.663439 27633 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.724370 28157 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:09.744586 27271 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:20:09.758067 28149 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20240426 13:20:09.760752 28148 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.761437 28148 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.763062 28148 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:09.764302 27498 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.743698 27653 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.767674 25026 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:09.715284 27725 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.757301 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.747408 27914 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:09.749955 27928 raft_consensus.cc:2741] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20240426 13:20:09.792467 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:09.732602 28155 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.038s
I20240426 13:20:09.796197 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.753504 27868 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.799122 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:09.792114 27914 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:09.807435 25026 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.808565 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.808451 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.809458 25104 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.810642 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.810777 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.811734 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.812196 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.812875 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.813222 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.813771 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.814524 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.814775 25103 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.815301 25026 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:09.816021 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.815336 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.817490 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.820189 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.820461 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.821139 25104 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.822155 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.824759 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 13:20:09.811749 25100 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:09.819808 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.829027 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.834111 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.832273 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.803526 27647 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.838281 27647 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.838747 27647 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:09.840374 28147 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20240426 13:20:09.794847 27271 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:09.805476 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:09.804984 27928 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:09.840550 28158 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
W20240426 13:20:09.867578 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62187.3us
I20240426 13:20:09.752305 27738 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
W20240426 13:20:09.985647 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 116235us
I20240426 13:20:09.840862 27914 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.046164 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.105235 25100 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:09.799679 28168 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:10.050099 27653 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:10.050228 27271 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.050592 28165 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.106958 27271 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.107105 28165 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.109125 27271 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.059712 28035 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.109925 28165 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.865339 28175 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:10.112946 28175 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:09.756124 28157 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.114288 27916 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.115073 28157 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:10.097949 28172 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.301s	user 0.000s	sys 0.000s
W20240426 13:20:10.098847 28156 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.366s	user 0.000s	sys 0.000s
W20240426 13:20:10.059908 28144 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20240426 13:20:10.111261 28035 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.132884 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.112098 27647 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.060242 28154 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.318s
I20240426 13:20:09.755218 28119 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.908692 27928 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.844530 27708 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.253s
W20240426 13:20:10.110602 28160 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.395s	user 0.001s	sys 0.000s
I20240426 13:20:10.218158 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:10.219028 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.062s	sys 0.104s
W20240426 13:20:10.228965 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.062s	sys 0.104s
W20240426 13:20:10.218569 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 541386us
I20240426 13:20:10.115093 28131 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.117244 28168 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.233803 25100 raft_consensus.cc:2396] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:10.230335 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.239166 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 489508us
I20240426 13:20:10.242563 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:10.251989 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 435145us
I20240426 13:20:10.134344 27653 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:10.206566 28034 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.264951 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2116) took 3029 ms. Trace:
I20240426 13:20:10.269601 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.217664 27410 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.229805 27928 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.274195 25030 rpcz_store.cc:276] 0426 13:20:07.235459 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.614717 (+1379258us) service_pool.cc:225] Handling call
0426 13:20:08.615041 (+   324us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.981907 (+366866us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.981953 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.982509 (+   556us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.538237 (+555728us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.538297 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.538403 (+   106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.264648 (+726245us) raft_consensus.cc:1671] finished
0426 13:20:10.264678 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.264877 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":366345,"raft.queue_time_us":366665,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":89,"thread_start_us":555287,"threads_started":1,"wal-append.queue_time_us":663328}
W20240426 13:20:10.280336 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.026s	sys 0.088s
W20240426 13:20:10.257385 28158 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20240426 13:20:10.280058 28171 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:10.270448 28144 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20240426 13:20:10.283584 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.285411 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.026s	sys 0.088s
W20240426 13:20:10.286213 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55977us
I20240426 13:20:10.296691 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.298241 27708 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.298674 27708 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.299105 27708 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.287701 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.300313 28171 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.302709 28171 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.299703 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.303458 28171 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.304284 28178 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:10.304905 28178 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:10.305969 28178 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:10.285812 28131 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.253680 28168 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:10.293270 27498 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.328645 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.329381 25622 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:10.331923 25100 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:10.333307 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.000s	sys 0.017s
W20240426 13:20:10.333637 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.000s	sys 0.017s
I20240426 13:20:10.334249 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2046) took 1097 ms. Trace:
I20240426 13:20:10.334578 24954 rpcz_store.cc:276] 0426 13:20:09.236583 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.463369 (+226786us) service_pool.cc:225] Handling call
0426 13:20:09.474249 (+ 10880us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.474430 (+   181us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:09.635235 (+160805us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.333961 (+698726us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:10.334008 (+    47us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:10.334049 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.334109 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.334207 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":160401,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":3068,"thread_start_us":698269,"threads_started":1}
I20240426 13:20:10.290113 27868 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.308487 28131 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.342972 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:10.328768 28062 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.732s	user 0.009s	sys 0.004s
I20240426 13:20:10.337989 28171 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.347780 28173 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:10.347776 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.346272 28159 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.401335 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.402052 24954 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:10.402642 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1705) took 1653 ms (client timeout 2023 ms). Trace:
I20240426 13:20:10.402980 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.398674 27725 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:10.403079 24951 rpcz_store.cc:269] 0426 13:20:08.749117 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.936254 (+187137us) service_pool.cc:225] Handling call
0426 13:20:10.402453 (+1466199us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:10.402586 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":937559808}
I20240426 13:20:10.345440 27653 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.417557 28035 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.403900 27725 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.405037 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:10.405081 27914 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.380510 28173 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.407692 24953 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:10.411088 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 616) took 1105 ms. Trace:
I20240426 13:20:10.426618 25617 rpcz_store.cc:276] 0426 13:20:09.305370 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.305801 (+   431us) service_pool.cc:225] Handling call
0426 13:20:09.306072 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.606993 (+300921us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.607035 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.607090 (+    55us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.607206 (+   116us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.410852 (+803646us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b5400473954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.410897 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.411031 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21017,"raft.queue_time_us":98709,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6777,"spinlock_wait_cycles":105480960,"thread_start_us":193624,"threads_started":1}
I20240426 13:20:10.419488 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2045) took 1183 ms. Trace:
I20240426 13:20:10.429018 24949 rpcz_store.cc:276] 0426 13:20:09.236347 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.456314 (+219967us) service_pool.cc:225] Handling call
0426 13:20:09.487120 (+ 30806us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.620367 (+133247us) raft_consensus.cc:1520] Early marking committed up to index 1
0426 13:20:09.620405 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.627898 (+  7493us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.229388 (+601490us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:20:10.229437 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.229548 (+   111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.419261 (+189713us) raft_consensus.cc:1671] finished
0426 13:20:10.419283 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.419417 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":132779,"raft.queue_time_us":142670,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":561563,"threads_started":1,"wal-append.queue_time_us":679133}
I20240426 13:20:10.431613 24949 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:10.434195 28168 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.436841 28173 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.438385 28173 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:10.440642 28168 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.442044 27914 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.443941 28035 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.444509 24949 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:10.447041 24951 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:10.448988 24949 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:10.456600 28131 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.455261 27725 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:10.464816 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 177928us
I20240426 13:20:10.467603 28096 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:10.468154 28096 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:10.469142 28096 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.470655 28096 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:10.472999 28155 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:10.473474 28155 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.474433 28155 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.481142 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:10.481920 24953 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.485502 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 442) took 2483 ms. Trace:
I20240426 13:20:10.485991 25618 rpcz_store.cc:276] 0426 13:20:08.001735 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.004654 (+  2919us) service_pool.cc:225] Handling call
0426 13:20:08.117110 (+112456us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.356909 (+1239799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.485213 (+1128304us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.485429 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1466532608}
I20240426 13:20:10.487936 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2132) took 1487 ms. Trace:
I20240426 13:20:10.488420 25025 rpcz_store.cc:276] 0426 13:20:09.000707 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.133056 (+132349us) service_pool.cc:225] Handling call
0426 13:20:10.487698 (+1354642us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:10.487857 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1029748608}
W20240426 13:20:10.493764 28170 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.382s	user 0.001s	sys 0.006s
I20240426 13:20:10.500429 28059 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:10.500912 28059 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.501972 28059 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.508224 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.06032e+06us
I20240426 13:20:10.458868 28164 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.448077 28035 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:10.517866 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51740.5us
I20240426 13:20:10.527087 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2051) took 1215 ms. Trace:
I20240426 13:20:10.526945 28175 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.527602 24952 rpcz_store.cc:276] 0426 13:20:09.312003 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.469868 (+157865us) service_pool.cc:225] Handling call
0426 13:20:09.470158 (+   290us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.590413 (+120255us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.590458 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.591101 (+   643us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.662609 (+ 71508us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.662663 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.662756 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.493264 (+830508us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b2400274088. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d247ce4 00007fda8d245dfb 00007fda8d2d3e09 00007fda8d2d3a95 00007fda8d2d30dd 00007fda8ff822ee 00007fda8ff99a8c 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1
0426 13:20:10.521718 (+ 28454us) raft_consensus.cc:1671] finished
0426 13:20:10.521747 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.527012 (+  5265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120884,"raft.queue_time_us":149771,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5985,"spinlock_wait_cycles":39615104,"thread_start_us":13900,"threads_started":1,"wal-append.queue_time_us":208108}
I20240426 13:20:10.517666 28059 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.421885 28159 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.519037 28155 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:10.538573 28175 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:10.549113 28161 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:10.549664 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 555) took 2570 ms. Trace:
I20240426 13:20:10.558085 25099 rpcz_store.cc:276] 0426 13:20:07.978835 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.555421 (+576586us) service_pool.cc:225] Handling call
0426 13:20:08.555730 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.216481 (+660751us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.549417 (+1332936us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400168854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.549599 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1517808000}
I20240426 13:20:10.561517 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.560242 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:10.577529 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:10.578161 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 294628us
I20240426 13:20:10.578305 25101 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:10.581142 25104 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:10.587528 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.588989 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.589870 25030 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:10.591048 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.566079 27438 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.593418 28159 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:10.593855 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 79620.9us
I20240426 13:20:10.594758 28141 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.596486 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:10.566810 28104 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.939s	user 0.008s	sys 0.002s
I20240426 13:20:10.556373 28047 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.599344 25096 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:10.599453 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.623193 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.624042 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.642618 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:10.626068 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.628484 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.629216 27438 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.579761 28173 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:10.631482 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.584556 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:10.555825 24953 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:20:10.694380 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109914us
I20240426 13:20:10.695410 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:10.621524 28036 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.566465 27928 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.630318 27810 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:10.727275 27928 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.726998 27810 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.728322 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92589.5us
I20240426 13:20:10.730263 27810 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.730938 27810 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:10.639405 27755 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.732254 24954 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:10.731973 27755 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.734450 27755 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.736378 27438 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.690313 25030 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:10.692916 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.741252 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.598220 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2031) took 1403 ms. Trace:
I20240426 13:20:10.743047 24950 rpcz_store.cc:276] 0426 13:20:09.194308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.428575 (+234267us) service_pool.cc:225] Handling call
0426 13:20:09.428860 (+   285us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:09.443909 (+ 15049us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:09.463310 (+ 19401us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.332824 (+869514us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:10.342928 (+ 10104us) log.cc:842] Serialized 52 byte log entry
0426 13:20:10.343093 (+   165us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:10.596722 (+253629us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.597257 (+   535us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.597450 (+   193us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.597528 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.597587 (+    59us) raft_consensus.cc:1671] finished
0426 13:20:10.597605 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.597778 (+   173us) inbound_call.cc:173] Queueing success response
0426 13:20:10.644503 (+ 46725us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
Metrics: {"mutex_wait_us":15919,"prepare.queue_time_us":154358,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":107,"raft.queue_time_us":30604,"raft.run_cpu_time_us":12343,"raft.run_wall_time_us":1249635,"spinlock_wait_cycles":2915200,"thread_start_us":924256,"threads_started":2,"wal-append.queue_time_us":804428}
I20240426 13:20:10.625576 27900 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.698426 25104 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:10.700135 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.640509 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.747660 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 107220us
I20240426 13:20:10.587635 28188 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20240426 13:20:10.748800 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.738925 28036 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.749984 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.751206 28188 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:10.639855 28104 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.716158 28193 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:10.752152 27647 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.752256 28193 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.752843 28193 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:10.752914 27647 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.687834 28047 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.753329 28104 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.753628 27647 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.754850 28047 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.754866 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.686751 28190 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.724941 28184 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.755720 28190 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.756258 28035 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.756424 28184 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.757795 28190 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.718482 28173 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.751628 28188 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.758523 28184 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.724723 27708 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.759181 28173 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.759385 28188 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.752864 28185 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240426 13:20:10.760556 28188 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.760600 27708 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.597060 27847 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20240426 13:20:10.753080 28176 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:10.557178 28164 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.761790 28176 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.762053 27847 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:10.762403 27847 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.762662 28176 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.718294 28179 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20240426 13:20:10.763185 27847 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.763674 28164 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.643945 27725 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.765497 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.765633 24953 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:10.766403 25099 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:10.766975 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.767200 24953 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:10.726085 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.769387 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.771320 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65447.3us
I20240426 13:20:10.772997 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:10.774755 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.011s	sys 0.024s
W20240426 13:20:10.776335 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.012s	sys 0.024s
I20240426 13:20:10.775771 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.785705 25617 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:10.779031 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.789963 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.791019 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.778930 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.717219 28062 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.794229 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.798875 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.630745 28025 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.135s
I20240426 13:20:10.644587 28159 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.808437 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.808665 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.780376 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.799988 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.774417 28168 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.808166 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.816185 24950 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:10.810827 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.778594 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.813194 25101 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:10.816748 25096 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:10.774609 28124 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:10.764364 27725 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.790287 27647 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:10.797148 28076 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:20:10.644956 27868 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.821910 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.833501 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.777896 27928 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.827425 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:10.835611 25031 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:10.834370 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.837885 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.839381 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.640916 28141 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.819855 27847 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:10.837057 28034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.028s
I20240426 13:20:10.862988 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.848214 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.24937e+06us
W20240426 13:20:10.856511 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83615.5us
I20240426 13:20:10.826782 27900 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.779220 28175 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20240426 13:20:10.860728 25030 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:10.892398 25022 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:10.861150 24950 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:10.863062 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.866786 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:10.894237 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:10.896281 27708 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.896441 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2029) took 1702 ms. Trace:
I20240426 13:20:10.896703 27708 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.897147 27708 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.900589 28178 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:10.901120 28178 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:10.901515 28178 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:10.896868 24955 rpcz_store.cc:276] 0426 13:20:09.193774 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.407113 (+213339us) service_pool.cc:225] Handling call
0426 13:20:09.424843 (+ 17730us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.425262 (+   419us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:20:09.425596 (+   334us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.633794 (+208198us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.643538 (+  9744us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.643696 (+   158us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.643906 (+   210us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.477159 (+833253us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:10.517907 (+ 40748us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:10.556430 (+ 38523us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:10.896130 (+339700us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.896208 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.896251 (+    43us) raft_consensus.cc:1671] finished
0426 13:20:10.896266 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.896387 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20883,"prepare.queue_time_us":4815,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"raft.queue_time_us":73163,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":32969,"spinlock_wait_cycles":37390720,"thread_start_us":526033,"threads_started":2,"wal-append.queue_time_us":216036}
I20240426 13:20:10.905268 28104 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:10.908466 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:10.910487 27498 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.910976 27498 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.911422 27498 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.868580 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.912720 27438 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:10.912878 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44410.6us
I20240426 13:20:10.913151 27438 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:10.913538 27438 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.913674 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:10.914865 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.916476 28025 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.916997 28025 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.923393 28025 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:10.862691 28034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.011s	sys 0.028s
I20240426 13:20:10.847841 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:10.928678 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:10.808403 28062 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.851152 27647 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:10.846411 28182 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.854106 28036 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.861729 28124 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.834936 28059 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20240426 13:20:10.938593 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75614.1us
I20240426 13:20:10.941829 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:10.869943 27653 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.957892 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:10.944581 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96833us
I20240426 13:20:10.928591 28097 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:10.822697 28159 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.945876 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.955595 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.929255 28061 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.003s	sys 0.027s
I20240426 13:20:10.967728 27438 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.937494 28139 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.846546 27868 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.940513 28152 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:10.970997 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.937956 27900 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.963660 28161 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.952200 27810 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:10.957258 25616 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:11.001439 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.976889 28061 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.003s	sys 0.028s
I20240426 13:20:10.977612 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.985383 28096 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.001s
I20240426 13:20:10.999399 27438 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.002008 25100 raft_consensus.cc:2396] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:10.956630 28035 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.022629 28035 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.981415 27647 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.024335 27647 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.026697 27647 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.027777 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.029407 28035 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.004179 25023 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:11.007131 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.981562 28175 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:10.960659 28036 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20240426 13:20:11.057749 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.000075 27810 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:11.068707 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 159164us
I20240426 13:20:11.015301 28195 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.001087 28206 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:11.083345 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:11.011518 27755 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 13:20:11.001367 28173 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:11.113698 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.114574 25102 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:11.087356 28204 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
W20240426 13:20:11.088424 28199 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20240426 13:20:11.092080 27498 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.100085 28159 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.100709 27725 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.069211 27708 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.083969 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.996002 27900 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.071713 27810 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:11.122246 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 156269us
I20240426 13:20:11.083052 28061 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.156085 25030 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.156762 25030 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.170068 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.063933 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.167110 25030 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:11.170118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 136151us
I20240426 13:20:11.173345 25030 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.149267 28061 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.085222 28195 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.212491 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.157155 28141 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.216384 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.165891 28062 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.213583 28139 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
W20240426 13:20:11.226650 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86663.4us
I20240426 13:20:11.209677 28059 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.244863 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.287846 28197 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:11.287849 28059 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.288585 28197 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.288923 28059 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.289194 28197 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:11.306846 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.309203 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.306846 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.310655 25031 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.313987 25025 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.216106 27868 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:11.317052 28193 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.564s	user 0.012s	sys 0.000s
I20240426 13:20:11.317129 27438 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:11.317752 28193 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.319536 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 617) took 2000 ms. Trace:
I20240426 13:20:11.320225 27708 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.321761 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 564) took 1077 ms. Trace:
I20240426 13:20:11.319977 25620 rpcz_store.cc:276] 0426 13:20:09.319137 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.319273 (+   136us) service_pool.cc:225] Handling call
0426 13:20:10.478998 (+1159725us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:10.479239 (+   241us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:10.546471 (+ 67232us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:10.546518 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:10.547031 (+   513us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.994131 (+447100us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:10.999027 (+  4896us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.999501 (+   474us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.999631 (+   130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:11.319248 (+319617us) raft_consensus.cc:1671] finished
0426 13:20:11.319353 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:11.319482 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44740,"prepare.queue_time_us":4275,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":49566,"raft.run_cpu_time_us":7220,"raft.run_wall_time_us":9764,"spinlock_wait_cycles":16928000,"thread_start_us":446681,"threads_started":1,"wal-append.queue_time_us":769862}
I20240426 13:20:11.322168 24948 rpcz_store.cc:276] 0426 13:20:10.244867 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.245081 (+   214us) service_pool.cc:225] Handling call
0426 13:20:10.245485 (+   404us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:10.268552 (+ 23067us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:10.474195 (+205643us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.829917 (+355722us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.831515 (+  1598us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:10.832025 (+   510us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.832191 (+   166us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:10.854256 (+ 22065us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.859748 (+  5492us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.859845 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:11.321590 (+461745us) raft_consensus.cc:1671] finished
0426 13:20:11.321609 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:11.321708 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31177,"prepare.queue_time_us":221048,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":70,"raft.queue_time_us":55364,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":44584,"spinlock_wait_cycles":37149824,"thread_start_us":549016,"threads_started":2,"wal-append.queue_time_us":468032}
W20240426 13:20:11.327384 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 111712us
I20240426 13:20:11.340243 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:11.341156 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 618) took 2006 ms (client timeout 1977 ms). Trace:
I20240426 13:20:11.341742 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 569) took 1086 ms. Trace:
W20240426 13:20:11.341545 25619 rpcz_store.cc:269] 0426 13:20:09.341009 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.348383 (+  7374us) service_pool.cc:225] Handling call
0426 13:20:11.340079 (+1991696us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:11.341087 (+  1008us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13832960}
I20240426 13:20:11.342250 24954 rpcz_store.cc:276] 0426 13:20:10.256909 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.435245 (+178336us) service_pool.cc:225] Handling call
0426 13:20:10.435572 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.732691 (+297119us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.341534 (+608843us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:11.341690 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":681811200}
I20240426 13:20:11.355154 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.356839 25103 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:11.358534 28195 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.359616 25097 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:11.362767 25025 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:11.365612 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:11.356211 27923 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:11.366887 28215 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
W20240426 13:20:11.367220 28207 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20240426 13:20:11.369597 28139 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:11.372553 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.351423 28193 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.375500 27853 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.376328 28237 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.377274 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 406) took 3919 ms. Trace:
I20240426 13:20:11.377671 25029 rpcz_store.cc:276] 0426 13:20:07.458336 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.239402 (+1781066us) service_pool.cc:225] Handling call
0426 13:20:09.468297 (+228895us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:09.489039 (+ 20742us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.377091 (+1888052us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54003f7d50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.377220 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":525655936}
I20240426 13:20:11.379552 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:11.381767 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.384543 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.385831 25030 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:20:11.380123 27498 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.401710 25097 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.322207 28178 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.411464 28178 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.399729 28237 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.414212 28237 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.415270 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.417886 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:11.403187 27847 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.000s
I20240426 13:20:11.429503 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.430570 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:11.430770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121666us
I20240426 13:20:11.389889 28139 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.432729 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.463460 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.463912 25622 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:11.465741 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.468235 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.414201 27900 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.427286 28227 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.475024 28197 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.477246 25028 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:11.477623 25028 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.478099 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.478694 25026 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.478359 25028 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.479463 25028 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.476879 28059 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.483186 28059 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.486879 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.488718 28197 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:11.489585 28163 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.428s	user 0.006s	sys 0.000s
I20240426 13:20:11.496884 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.497596 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:11.497897 28216 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
I20240426 13:20:11.498582 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.499230 24949 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:11.500591 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 418494us
I20240426 13:20:11.433394 28139 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.503289 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.504258 24952 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.506477 24949 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.507923 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.508567 24953 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:11.509064 25102 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:11.512319 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.512909 24952 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.513809 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.514422 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.546463 24949 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:11.518306 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.550963 24953 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:11.502120 28131 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.015s	sys 0.041s
I20240426 13:20:11.554283 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.554905 24949 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.555670 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.426983 28193 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
W20240426 13:20:11.546864 28188 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.786s	user 0.008s	sys 0.004s
I20240426 13:20:11.558194 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.557741 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:11.557315 28131 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.006s	user 0.016s	sys 0.041s
I20240426 13:20:11.433576 28212 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
I20240426 13:20:11.508352 27928 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.503588 28237 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.508857 27799 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:11.609545 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.610373 24955 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:11.563267 25028 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.565779 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:11.520514 25026 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.502900 27900 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.567114 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.616403 24953 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:11.616593 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49587.7us
I20240426 13:20:11.549696 27932 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.617368 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.618019 24953 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.618835 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.619514 24953 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.517165 28168 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.548920 28197 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.620839 28168 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.621662 28197 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.508985 28139 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.604709 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.624289 28176 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:11.625252 27853 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:20:11.562602 28194 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 13:20:11.553557 27810 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.630033 27810 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.630770 25024 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:11.545428 28059 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:20:11.614900 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.623267 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:11.613515 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2156) took 1034 ms. Trace:
I20240426 13:20:11.632107 28059 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.632614 28059 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:11.632369 25028 rpcz_store.cc:276] 0426 13:20:10.579506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.579678 (+   172us) service_pool.cc:225] Handling call
0426 13:20:10.580180 (+   502us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:11.477150 (+896970us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b54002894a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:20:11.613211 (+136061us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.613449 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158845696}
I20240426 13:20:11.633855 28059 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:11.635774 27943 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20240426 13:20:11.613380 27928 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.568955 28126 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.609027 27799 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:11.645866 24948 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:11.648808 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.612728 28025 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.469471 28239 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.181s
I20240426 13:20:11.658607 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:11.659174 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 226525us
I20240426 13:20:11.660122 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.653040 24953 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:11.661108 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.661810 24955 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.644655 28193 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.667901 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.667541 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.668073 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.662444 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.671483 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.672279 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:11.680684 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 211017us
I20240426 13:20:11.681862 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.683300 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.684705 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.686058 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.686133 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.692770 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.693378 24953 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.666460 27810 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.614512 28237 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.708549 24955 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:11.709746 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.711477 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.503259 28227 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.662011 27928 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.721648 25097 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:11.724812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.725152 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.651439 27900 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.694576 27869 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.691890 28059 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.725536 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:11.733662 24948 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:11.734556 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.735121 24948 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.697011 28212 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:11.736655 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.696322 28173 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:11.729189 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.696316 28034 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:11.730312 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.697010 28178 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:11.663372 28025 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.696388 27868 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:11.710729 27916 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.744993 27916 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.745388 27916 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.746652 27900 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.739898 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:11.748559 25097 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.749588 28222 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.749982 24948 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:11.696725 27498 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.740418 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.725908 28168 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.752185 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.772714 24953 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.779435 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.696875 28175 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20240426 13:20:11.682869 28193 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.789943 24953 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:11.744457 27451 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.731400 28224 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.700536 27810 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.700656 28161 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20240426 13:20:11.732340 27869 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.652415 28126 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.740449 28034 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.739898 28178 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.750180 28222 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.752384 27946 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:11.798617 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46518.8us
I20240426 13:20:11.751066 27498 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.754765 28237 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.720167 28166 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.070s
I20240426 13:20:11.810566 28222 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.816798 28025 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.800925 27451 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.817955 27451 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.800015 27810 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.821293 28243 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.821739 28243 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.822096 28243 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.804011 28034 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.824484 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.825377 25026 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:11.711624 27799 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.826964 27799 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.807986 28178 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.803674 28184 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.832692 28184 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.833604 28184 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.834626 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 638) took 1056 ms. Trace:
I20240426 13:20:11.835063 25622 rpcz_store.cc:276] 0426 13:20:10.778242 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.778364 (+   122us) service_pool.cc:225] Handling call
0426 13:20:11.834471 (+1056107us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.834574 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1485646208}
I20240426 13:20:11.836897 28227 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.838750 28126 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.804255 27869 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.841033 28193 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.841328 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.842504 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.843051 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:11.812393 28197 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.814325 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.739300 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:11.767555 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1537) took 1144 ms. Trace:
I20240426 13:20:11.870800 25100 rpcz_store.cc:276] 0426 13:20:10.622648 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.622841 (+   193us) service_pool.cc:225] Handling call
0426 13:20:11.767322 (+1144481us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b540001bdd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.767481 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":722548480}
I20240426 13:20:11.872107 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.750802 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.873462 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.750944 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.874130 24955 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:11.874239 24948 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:11.752660 28233 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20240426 13:20:11.875799 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.876863 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.877960 25028 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:11.868399 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.868649 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.843725 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:11.763958 28202 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.768s	user 0.000s	sys 0.002s
I20240426 13:20:11.807346 25097 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:11.813324 27946 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.882719 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.815515 27498 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.880532 25102 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:20:11.903520 24948 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.904222 25100 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240426 13:20:11.888352 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.767738 27868 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.765384 28240 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.916734 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:11.884858 28126 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:11.926095 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.923487 26679 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:11.887287 27725 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.905498 27932 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:11.927572 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.761216 27916 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.800573 28224 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.914932 27911 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:11.814669 28247 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.114s
I20240426 13:20:11.917048 28240 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.952889 27911 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.953498 27911 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.954223 28025 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.954767 28025 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.921268 28222 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:11.957257 28246 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20240426 13:20:11.912220 27868 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.923059 27738 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.906205 28035 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20240426 13:20:11.965173 25102 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:11.953097 28247 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:11.965685 28247 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.966607 28247 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.967191 28034 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.887754 28055 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:11.906214 27946 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.971360 27869 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.925101 28097 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:11.972481 28097 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.972856 28097 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:11.978354 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 575) took 1416 ms. Trace:
I20240426 13:20:11.978699 28224 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.978749 25099 rpcz_store.cc:276] 0426 13:20:10.562088 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.562703 (+   615us) service_pool.cc:225] Handling call
0426 13:20:10.562965 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.766867 (+203902us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.978131 (+1211264us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:11.978286 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467356928}
I20240426 13:20:11.924918 28126 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.982947 28126 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.964628 27738 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:11.988487 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 249667us
I20240426 13:20:11.990602 25615 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:11.951021 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:11.999034 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 619) took 2650 ms (client timeout 1891 ms). Trace:
W20240426 13:20:11.999539 25615 rpcz_store.cc:269] 0426 13:20:09.348958 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.349111 (+   153us) service_pool.cc:225] Handling call
0426 13:20:11.565624 (+2216513us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:11.998923 (+433299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12372352}
I20240426 13:20:11.966500 28175 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.926621 28212 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.020077 24956 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:12.022804 24954 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:12.024708 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2048) took 2770 ms (client timeout 1977 ms). Trace:
W20240426 13:20:12.025259 24947 rpcz_store.cc:269] 0426 13:20:09.247119 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.263480 (+ 16361us) service_pool.cc:225] Handling call
0426 13:20:12.017768 (+2754288us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:12.017917 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78129024}
I20240426 13:20:12.027160 24947 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:12.028924 24947 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:11.874841 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:12.030761 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 294180us
I20240426 13:20:12.024708 24954 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:12.060598 25105 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:12.192023 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:12.193768 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:12.194790 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 284688us
I20240426 13:20:11.995870 27725 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:12.209962 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.209745 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.216872 25028 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:12.218391 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.218591 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:12.219031 25028 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:12.018088 28096 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.200873 24955 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:12.200234 25619 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:12.209471 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.958647 28240 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:12.197521 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 134345us
I20240426 13:20:12.212383 27869 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.190057 28247 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:12.196766 27271 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.221025 28238 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:12.222187 28233 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:12.224788 28047 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:12.226691 28241 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.749s	user 0.003s	sys 0.000s
I20240426 13:20:12.229775 27932 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:20:12.231081 28139 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.608s	user 0.013s	sys 0.002s
I20240426 13:20:12.232460 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1162) took 1292 ms. Trace:
I20240426 13:20:12.288682 27799 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:12.244985 28249 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20240426 13:20:12.246099 27738 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.248994 27410 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:12.250559 27911 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.252156 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.196640 28168 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.254348 28178 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.210516 28212 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.213644 27708 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:12.196702 28049 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.256316 27946 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.213292 28188 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.060817 28171 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.271641 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.197908 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:12.346909 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 149459us
I20240426 13:20:12.347929 24995 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:12.348269 24995 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:12.349103 25097 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:12.287418 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.350596 25024 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.023947 28175 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.292881 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:12.259543 28222 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.220160 27923 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20240426 13:20:12.163779 28243 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.236457 28142 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.306525 25101 rpcz_store.cc:276] 0426 13:20:10.939800 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.939956 (+   156us) service_pool.cc:225] Handling call
0426 13:20:12.231757 (+1291801us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:12.232379 (+   622us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1083905280}
I20240426 13:20:12.329960 24949 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:12.262223 28096 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.303315 28025 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.311913 27410 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:12.345909 24955 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:12.331070 27932 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.300004 27271 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.354723 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.380115 25099 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:12.312587 27869 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:12.365727 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.381711 24955 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:12.357955 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65162.3us
I20240426 13:20:12.382961 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:12.354516 25103 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:12.379998 24950 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.384495 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:12.384771 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.306478 28047 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.306054 27810 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.359871 28055 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:12.359264 27923 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:12.308521 28139 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.396235 24949 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:12.402094 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.357507 28254 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.421978 24955 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:12.452298 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 7, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 7 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:12.423282 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 383898us
I20240426 13:20:12.406956 24947 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:12.352917 28258 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:12.428412 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.459051 24949 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:12.428530 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:12.311831 27799 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.365725 28096 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.365109 27410 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.364449 28025 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:12.460829 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:12.461136 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 566) took 2213 ms. Trace:
I20240426 13:20:12.387269 28148 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:12.433918 27911 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.395749 28154 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:12.434792 28178 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.435446 28124 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:12.359707 28142 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.437166 27350 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.439122 27946 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.360517 28243 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.356158 28175 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.353938 27868 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.427467 27923 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.444718 28139 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.348934 28049 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.351406 28188 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.447649 27271 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.383793 27869 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:12.338014 28168 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.376363 28257 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:12.351161 28171 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.422827 28055 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:12.456144 28097 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:12.600203 28256 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20240426 13:20:12.635542 28264 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20240426 13:20:12.468751 28155 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:12.468508 28154 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.514452 28258 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.813068 28269 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:12.849442 28154 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:12.869872 28269 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:12.809427 28155 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.871098 28155 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:12.873421 28168 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.456445 28254 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.871223 28269 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:12.907199 28262 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20240426 13:20:12.911033 28269 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:12.457369 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 186295us
W20240426 13:20:12.457120 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.780s (SENT) [suppressed 22 similar messages]
W20240426 13:20:12.950306 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 492993us
I20240426 13:20:12.960603 24950 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:12.461504 24951 rpcz_store.cc:276] 0426 13:20:10.247269 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.419217 (+171948us) service_pool.cc:225] Handling call
0426 13:20:10.423314 (+  4097us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.447416 (+ 24102us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:12.460930 (+2013514us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:12.461086 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54131072}
I20240426 13:20:12.455122 24955 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20240426 13:20:12.968581 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1715) took 1285 ms. Trace:
I20240426 13:20:12.968449 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.969300 24955 rpcz_store.cc:276] 0426 13:20:11.682570 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.454525 (+771955us) service_pool.cc:225] Handling call
0426 13:20:12.454887 (+   362us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:12.967696 (+512809us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:12.968348 (+   652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:12.971004 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.971248 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.972213 24951 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.985086 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:12.445400 25023 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:12.521301 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.472131 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.995191 25022 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:12.995919 25027 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:12.996491 25025 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:12.997025 25025 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.997668 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.998101 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.998602 25025 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.515002 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.000005 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:12.515523 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 302047us
I20240426 13:20:13.001441 25031 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:12.996862 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.995834 25022 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.004802 25022 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.995095 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.006085 25022 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.008011 25030 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:13.009765 25025 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:12.309597 27725 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.467088 28096 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.067140 28161 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.067344 28139 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.773460 28047 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.454784 28126 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.088325 28055 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.090685 28055 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:12.788970 27923 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.092473 28175 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.076751 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1849) took 1388 ms. Trace:
I20240426 13:20:13.093475 28175 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.093864 25105 rpcz_store.cc:276] 0426 13:20:11.687896 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.688085 (+   189us) service_pool.cc:225] Handling call
0426 13:20:11.711277 (+ 23192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.193498 (+482221us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.075938 (+882440us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.076435 (+   497us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":787088640}
I20240426 13:20:13.096117 28168 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.096943 28168 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.101373 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:13.100734 28258 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.101536 27869 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.103256 27869 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:13.103627 27868 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.088752 25100 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:13.104441 28209 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.105152 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.105556 28209 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.107583 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.108310 27799 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.525749 28025 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.708926 27708 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:12.467224 27932 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.722608 27810 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:12.465270 28059 consensus_queue.cc:1159] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20240426 13:20:13.076895 28097 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.836330 28124 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.200695 28097 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.202397 28097 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.133232 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.108536 25030 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:13.206739 28139 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.134836 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 579) took 2281 ms. Trace:
I20240426 13:20:13.139349 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1852) took 1412 ms. Trace:
W20240426 13:20:13.112262 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127757us
I20240426 13:20:13.114872 28257 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:13.131417 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.107599 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:13.119269 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49506.7us
I20240426 13:20:13.116880 24950 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20240426 13:20:13.143826 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 141971us
I20240426 13:20:13.113945 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1734) took 1378 ms. Trace:
I20240426 13:20:13.134212 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.099871 28168 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.138139 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.288408 24953 rpcz_store.cc:276] 0426 13:20:11.735466 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.779118 (+ 43652us) service_pool.cc:225] Handling call
0426 13:20:13.113404 (+1334286us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.113700 (+   296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22328192}
I20240426 13:20:13.132427 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.304203 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1719) took 1595 ms. Trace:
I20240426 13:20:13.114598 28248 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:13.313668 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.313956 28248 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.314517 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.315353 28139 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.109059 28206 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.316485 28206 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:13.185107 27451 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.308666 24950 rpcz_store.cc:276] 0426 13:20:11.701597 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.115815 (+1414218us) service_pool.cc:225] Handling call
0426 13:20:13.116482 (+   667us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.291100 (+174618us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.297258 (+  6158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.291517 25096 rpcz_store.cc:276] 0426 13:20:10.853530 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.853976 (+   446us) service_pool.cc:225] Handling call
0426 13:20:10.865672 (+ 11696us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.993857 (+128185us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:11.187286 (+193429us) log.cc:842] Serialized 14 byte log entry
0426 13:20:11.961399 (+774113us) spinlock_profiling.cc:243] Waited 5.24 ms on lock 0x7b18003996a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:20:12.198612 (+237213us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:12.198684 (+    72us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:12.198745 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.134197 (+935452us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.134280 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.134601 (+   321us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":205454,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":1962,"raft.queue_time_us":12932,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":45598,"spinlock_wait_cycles":306289792,"thread_start_us":393690,"threads_started":2,"wal-append.queue_time_us":1093287}
I20240426 13:20:13.109306 28237 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.295562 28257 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.201357 28049 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.292061 27708 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.324232 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.111857 28254 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.114203 28243 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.309996 25102 rpcz_store.cc:276] 0426 13:20:11.726932 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.902444 (+175512us) service_pool.cc:225] Handling call
0426 13:20:11.909396 (+  6952us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:11.965600 (+ 56204us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.138733 (+1173133us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.139179 (+   446us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127659776}
I20240426 13:20:13.342885 25105 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.115526 28096 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.351830 25620 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.326319 28257 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:13.200062 28124 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.414465 28221 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 2.066s	user 0.000s	sys 0.016s
W20240426 13:20:13.372288 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46436.1us
I20240426 13:20:13.415669 28124 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.376890 24950 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20240426 13:20:13.435803 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1720) took 1712 ms. Trace:
I20240426 13:20:13.435914 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1841) took 2434 ms. Trace:
I20240426 13:20:13.436292 24950 rpcz_store.cc:276] 0426 13:20:11.723679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.328268 (+1604589us) service_pool.cc:225] Handling call
0426 13:20:13.376581 (+ 48313us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.435469 (+ 58888us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.435717 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.436358 25104 rpcz_store.cc:276] 0426 13:20:11.001299 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.001441 (+   142us) service_pool.cc:225] Handling call
0426 13:20:11.001704 (+   263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:11.929571 (+927867us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.435429 (+1505858us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.435717 (+   288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119262976}
I20240426 13:20:13.437893 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.328003 28237 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.380496 28142 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.381181 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.382392 25096 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:13.382732 27916 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.388020 28173 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.389474 28174 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:13.392710 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:13.393222 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.017s	sys 0.040s
I20240426 13:20:13.393153 28206 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:13.393190 28126 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.328086 27451 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.395720 27923 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.396337 25618 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:20:13.132382 24853 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 9, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 9 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:13.397733 28096 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:13.398223 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50291.5us
I20240426 13:20:13.398854 28097 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.399444 25103 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:13.337020 27799 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.400012 28195 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:13.140674 27350 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.331281 28175 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.402037 28257 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.408816 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.409132 28248 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.409523 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.363236 28209 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.372898 28243 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.416981 28225 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 2.060s	user 0.015s	sys 0.004s
I20240426 13:20:13.416747 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:13.426178 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1855) took 1368 ms. Trace:
I20240426 13:20:13.427847 28159 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.430259 28025 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.431442 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.432018 25022 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20240426 13:20:13.432489 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.438050 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.440639 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2091) took 1533 ms. Trace:
W20240426 13:20:13.441474 28155 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.570s	user 0.005s	sys 0.010s
I20240426 13:20:13.443096 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1858) took 1346 ms. Trace:
I20240426 13:20:13.299013 28161 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.444511 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 579) took 1254 ms. Trace:
I20240426 13:20:13.523150 24947 rpcz_store.cc:276] 0426 13:20:12.190490 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.207756 (+ 17266us) service_pool.cc:225] Handling call
0426 13:20:13.444200 (+1236444us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.444368 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313253248}
I20240426 13:20:13.524443 28271 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.524720 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.449120 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 557) took 4428 ms. Trace:
I20240426 13:20:13.525303 28271 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.525738 24956 rpcz_store.cc:276] 0426 13:20:09.020709 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.347614 (+326905us) service_pool.cc:225] Handling call
0426 13:20:12.018107 (+2670493us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.020675 (+  2568us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.448659 (+1427984us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b5400366ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.448893 (+   234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6140787456}
I20240426 13:20:13.526783 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.527616 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:13.528482 28286 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.528928 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.529291 28161 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.529095 28286 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:13.529922 28155 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.530635 25030 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:13.531402 28286 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.450249 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1850) took 1757 ms. Trace:
W20240426 13:20:13.450932 27677 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:20:13.532099 28155 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.532897 28155 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.532927 28159 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.533655 28155 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.535073 28237 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.535248 28159 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.535653 28155 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.458273 25025 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:13.469995 25105 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:13.536979 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1865) took 1074 ms. Trace:
I20240426 13:20:13.536861 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.537372 25105 rpcz_store.cc:276] 0426 13:20:12.462849 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.107072 (+644223us) service_pool.cc:225] Handling call
0426 13:20:13.536889 (+429817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.538252 27923 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.538516 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.538844 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.518822 25022 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:20:13.539868 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:13.478092 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s	user 0.018s	sys 0.041s
I20240426 13:20:13.462463 28168 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.542338 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.543413 25617 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:13.542938 28168 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.544106 25620 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:13.540200 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.545274 28168 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.503033 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1861) took 1262 ms. Trace:
I20240426 13:20:13.545543 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:13.449349 27498 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.533s	user 0.016s	sys 0.003s
I20240426 13:20:13.516001 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.512900 24953 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:13.528203 28152 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20240426 13:20:13.541836 28242 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20240426 13:20:13.541358 25100 rpcz_store.cc:276] 0426 13:20:12.097045 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.217019 (+119974us) service_pool.cc:225] Handling call
0426 13:20:12.217287 (+   268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.104492 (+887205us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.442831 (+338339us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.443047 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1983318784}
I20240426 13:20:13.468462 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1862) took 1199 ms. Trace:
I20240426 13:20:13.554407 25096 rpcz_store.cc:276] 0426 13:20:12.263918 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.381519 (+1117601us) service_pool.cc:225] Handling call
0426 13:20:13.382032 (+   513us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.463411 (+ 81379us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.463673 (+   262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.515942 25622 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.527540 24956 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20240426 13:20:13.514111 25099 rpcz_store.cc:276] 0426 13:20:12.058075 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.190431 (+132356us) service_pool.cc:225] Handling call
0426 13:20:12.190692 (+   261us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.380609 (+189917us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.425641 (+1045032us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b540001bdd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.425972 (+   331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":435160320}
I20240426 13:20:13.519575 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:13.546031 25103 rpcz_store.cc:276] 0426 13:20:12.235993 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.363183 (+1127190us) service_pool.cc:225] Handling call
0426 13:20:13.399128 (+ 35945us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.498448 (+ 99320us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.498719 (+   271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.545291 27350 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:13.540158 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 123641us
I20240426 13:20:13.546875 25022 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20240426 13:20:13.558063 25022 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:13.532199 25097 rpcz_store.cc:276] 0426 13:20:11.692789 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.901192 (+208403us) service_pool.cc:225] Handling call
0426 13:20:11.901612 (+   420us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.349717 (+448105us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.449769 (+1100052us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.450049 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1028578048}
I20240426 13:20:13.558809 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:13.558925 28281 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.559058 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:13.559957 25097 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:13.559713 28281 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.560413 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:13.561525 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:13.562273 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:13.564265 28097 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.565014 28097 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.565604 28279 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.566339 28279 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.567602 28257 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.568445 28161 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:13.569576 28152 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.570137 28152 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.570973 27498 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:13.539798 28154 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.607s	user 0.003s	sys 0.008s
W20240426 13:20:13.552799 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87001.1us
I20240426 13:20:13.549326 25104 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.560559 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1721) took 1829 ms. Trace:
I20240426 13:20:13.519150 24948 rpcz_store.cc:276] 0426 13:20:11.907048 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.920206 (+ 13158us) service_pool.cc:225] Handling call
0426 13:20:13.440422 (+1520216us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000be854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.440574 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2533716736}
I20240426 13:20:13.574169 24956 rpcz_store.cc:276] 0426 13:20:11.726553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.526790 (+1800237us) service_pool.cc:225] Handling call
0426 13:20:13.527227 (+   437us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.555819 (+ 28592us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.556071 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.539449 28025 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.571435 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2089) took 1906 ms. Trace:
I20240426 13:20:13.532191 27451 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.540264 28268 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.527930 28271 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.550545 28055 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:13.572166 28097 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.575052 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.577618 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.511276 27799 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.580381 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.582199 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1718) took 1880 ms. Trace:
I20240426 13:20:13.587819 25618 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:13.591009 24951 rpcz_store.cc:276] 0426 13:20:11.698081 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.972921 (+1274840us) service_pool.cc:225] Handling call
0426 13:20:12.974829 (+  1908us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.072967 (+ 98138us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.073013 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.207495 (+134482us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.293051 (+ 85556us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.507313 (+214262us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.576987 (+ 69674us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.577040 (+    53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.577068 (+    28us) raft_consensus.cc:1671] finished
0426 13:20:13.577076 (+     8us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.578573 (+  1497us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91609,"prepare.queue_time_us":125036,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":65,"raft.queue_time_us":586876,"raft.run_cpu_time_us":974,"raft.run_wall_time_us":48374,"thread_start_us":237382,"threads_started":3,"wal-append.queue_time_us":208578}
I20240426 13:20:13.597151 28165 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:13.597478 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.598109 28165 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:13.599099 24953 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [7.4-9.5]
I20240426 13:20:13.599817 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1722) took 1869 ms. Trace:
I20240426 13:20:13.598268 25618 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.582096 28154 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.600288 24953 rpcz_store.cc:276] 0426 13:20:11.730808 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.554407 (+1823599us) service_pool.cc:225] Handling call
0426 13:20:13.598692 (+ 44285us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:13.599546 (+   854us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.599742 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.603387 25619 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:13.604945 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95852.4us
I20240426 13:20:13.539449 28173 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.607609 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.571434 28152 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.518280 27708 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.587638 28055 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.607107 28025 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.614900 28025 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.616412 27946 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:13.617067 27946 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.617561 27946 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:13.619251 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.620805 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:13.622169 28161 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:13.623184 28161 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:13.623095 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:13.624326 28257 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.624604 27498 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.624742 28257 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.625020 24853 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:13.625852 28257 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.627305 28257 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.629448 27799 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.629810 25096 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:13.578248 24952 rpcz_store.cc:276] 0426 13:20:11.664762 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.105217 (+440455us) service_pool.cc:225] Handling call
0426 13:20:12.165316 (+ 60099us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.165575 (+   259us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:12.463783 (+298208us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.570934 (+1107151us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.571036 (+   102us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.571109 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.571215 (+   106us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.571365 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19821,"prepare.queue_time_us":1237492,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":20017,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":1337988,"threads_started":2,"wal-append.queue_time_us":872691}
I20240426 13:20:13.633176 24952 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:13.633455 27410 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:13.623221 28243 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.637048 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1724) took 1904 ms. Trace:
I20240426 13:20:13.636907 28293 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:13.637717 28281 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.638010 24947 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:13.638279 27868 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:13.629725 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.638918 27868 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:13.640082 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.638706 28154 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.640663 27350 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:13.649462 28248 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:13.642375 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:13.650632 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.651302 25102 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:13.651578 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1860) took 1425 ms. Trace:
I20240426 13:20:13.645431 24947 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:13.651784 27708 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.654276 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1733) took 1919 ms. Trace:
I20240426 13:20:13.654693 24947 rpcz_store.cc:276] 0426 13:20:11.735219 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.639540 (+1904321us) service_pool.cc:225] Handling call
0426 13:20:13.639925 (+   385us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.654018 (+ 14093us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.654200 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.656217 25026 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20240426 13:20:13.656752 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:13.637507 24956 rpcz_store.cc:276] 0426 13:20:11.732935 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.589054 (+1856119us) service_pool.cc:225] Handling call
0426 13:20:13.636421 (+ 47367us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.636722 (+   301us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.636748 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.636808 (+    60us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:13.636848 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.636893 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.637001 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.651086 28061 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.645417 25103 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:13.657022 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 588) took 1370 ms. Trace:
I20240426 13:20:13.642431 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:13.657616 25618 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:13.627070 24853 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:13.659446 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53621.5us
I20240426 13:20:13.657187 25101 rpcz_store.cc:276] 0426 13:20:12.226429 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.360944 (+134515us) service_pool.cc:225] Handling call
0426 13:20:12.361349 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.650660 (+1289311us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.651345 (+   685us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540001b3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.651525 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2814869760}
I20240426 13:20:13.658016 25100 rpcz_store.cc:276] 0426 13:20:12.286467 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.568174 (+1281707us) service_pool.cc:225] Handling call
0426 13:20:13.575347 (+  7173us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.583754 (+  8407us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.583789 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.583840 (+    51us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:13.583880 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.583943 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.656877 (+ 72934us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b54003ef8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.656972 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15490304}
I20240426 13:20:13.633740 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1732) took 1898 ms. Trace:
I20240426 13:20:13.660790 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1163) took 1241 ms. Trace:
I20240426 13:20:13.661007 24952 rpcz_store.cc:276] 0426 13:20:11.734955 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.632608 (+1897653us) service_pool.cc:225] Handling call
0426 13:20:13.632977 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.633515 (+   538us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.633675 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.661406 25096 rpcz_store.cc:276] 0426 13:20:12.407657 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.630748 (+1223091us) service_pool.cc:225] Handling call
0426 13:20:13.648799 (+ 18051us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.664968 25622 raft_consensus.cc:2362] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3. Re-sending same reply.
I20240426 13:20:13.646474 28055 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.663048 28237 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:13.697649 28237 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.699020 28237 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.684924 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2092) took 1777 ms. Trace:
I20240426 13:20:13.640107 27868 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.689713 28173 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.710551 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:13.711359 28281 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:13.711829 28281 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.712137 28188 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:13.691952 27869 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.588s	user 0.005s	sys 0.004s
I20240426 13:20:13.599658 28165 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.714509 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.715312 25615 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:13.653012 27943 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:13.716187 27943 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.691380 28061 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:13.701359 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 112221us
I20240426 13:20:13.705929 24947 rpcz_store.cc:276] 0426 13:20:11.907462 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.655518 (+1748056us) service_pool.cc:225] Handling call
0426 13:20:13.666128 (+ 10610us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.677722 (+ 11594us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.677761 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.684648 (+  6887us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.684693 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.684750 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.684876 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.709539 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.712189 28281 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.696522 28152 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.716836 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.701609 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:13.721622 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43581.9us
I20240426 13:20:13.712275 24948 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:13.588953 28268 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.700156 28269 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.754s	user 0.010s	sys 0.001s
I20240426 13:20:13.711436 28279 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.765614 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1729) took 2031 ms. Trace:
I20240426 13:20:13.765982 24948 rpcz_store.cc:276] 0426 13:20:11.734178 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.596580 (+1862402us) service_pool.cc:225] Handling call
0426 13:20:13.700498 (+103918us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.764350 (+ 63852us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.765549 (+  1199us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:13.766976 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 121606us
I20240426 13:20:13.790685 25105 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:13.720503 27943 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.794744 27932 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:13.795254 27932 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.795624 27932 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.799575 27677 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:13.800173 28153 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:13.800478 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 217373us
W20240426 13:20:13.801034 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 298684us
I20240426 13:20:13.801427 27677 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:13.802095 28142 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.802372 27677 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:13.802551 28142 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.802861 27677 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.804168 28295 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.804725 28295 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.805954 28195 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:13.806273 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.806944 25618 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Advancing to term 6
I20240426 13:20:13.807631 28034 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.789009 27350 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.800762 28243 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:13.724478 28055 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.831323 25615 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.832082 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.805984 28062 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:13.836838 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.838102 25024 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.839653 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.847740 25623 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.849046 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.850976 24947 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:13.803807 28142 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.851653 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2095) took 1943 ms. Trace:
I20240426 13:20:13.855170 24947 rpcz_store.cc:276] 0426 13:20:11.908380 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.762381 (+1854001us) service_pool.cc:225] Handling call
0426 13:20:13.826980 (+ 64599us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.851400 (+ 24420us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.851595 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1443712}
I20240426 13:20:13.789631 28248 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20240426 13:20:13.854256 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 580) took 1663 ms. Trace:
I20240426 13:20:13.856283 24948 rpcz_store.cc:276] 0426 13:20:12.190785 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.778051 (+1587266us) service_pool.cc:225] Handling call
0426 13:20:13.841247 (+ 63196us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.853838 (+ 12591us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.853877 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.853934 (+    57us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:13.853979 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.854084 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.854198 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.858774 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.859709 28055 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.859490 28281 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.861615 28281 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.865880 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2093) took 1958 ms. Trace:
W20240426 13:20:13.871120 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46529.7us
I20240426 13:20:13.836344 28096 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:13.837783 28062 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.876325 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.871608 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.866871 24956 rpcz_store.cc:276] 0426 13:20:11.907781 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.663416 (+1755635us) service_pool.cc:225] Handling call
0426 13:20:13.689493 (+ 26077us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.689787 (+   294us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:13.690412 (+   625us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.849320 (+158908us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.865052 (+ 15732us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.865326 (+   274us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:13.865458 (+   132us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.865663 (+   205us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.865826 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23115,"prepare.queue_time_us":22607,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":137117,"threads_started":1}
I20240426 13:20:13.886632 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1731) took 2152 ms. Trace:
I20240426 13:20:13.800228 28161 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.077s
I20240426 13:20:13.800834 28153 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.886760 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.891041 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.894258 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.898247 24947 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:13.898361 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.888980 24953 rpcz_store.cc:276] 0426 13:20:11.734687 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.611353 (+1876666us) service_pool.cc:225] Handling call
0426 13:20:13.639498 (+ 28145us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.639853 (+   355us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:13.658341 (+ 18488us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.886302 (+227961us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.886358 (+    56us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.886401 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.886468 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.886583 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":217,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":36,"raft.queue_time_us":10590,"thread_start_us":195302,"threads_started":1}
I20240426 13:20:13.881160 28062 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:13.896610 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54939.7us
I20240426 13:20:13.900460 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.900835 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.901043 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.904578 25025 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:13.904958 25617 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:20:13.905323 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139556us
I20240426 13:20:13.804033 27677 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:13.905784 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 76861us
I20240426 13:20:13.908579 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.909603 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.909852 27677 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:13.910306 25027 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:20:13.910698 28296 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20240426 13:20:13.911303 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.911958 28239 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20240426 13:20:13.913946 28141 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.914536 28141 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.916065 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1717) took 2142 ms. Trace:
I20240426 13:20:13.916301 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.916538 24949 rpcz_store.cc:276] 0426 13:20:11.692632 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.459636 (+767004us) service_pool.cc:225] Handling call
0426 13:20:12.459956 (+   320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:12.960067 (+500111us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:12.960163 (+    96us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:12.961457 (+  1294us) log.cc:842] Serialized 26 byte log entry
0426 13:20:12.993085 (+ 31628us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.300195 (+307110us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.834914 (+534719us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.834992 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.835043 (+    51us) raft_consensus.cc:1671] finished
0426 13:20:13.835056 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.835203 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82425,"prepare.queue_time_us":393381,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":37767,"raft.queue_time_us":578371,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":23731,"thread_start_us":648904,"threads_started":3,"wal-append.queue_time_us":459918}
I20240426 13:20:13.918690 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.920378 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.922307 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1723) took 2189 ms. Trace:
I20240426 13:20:13.923425 28254 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:13.924108 28153 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.924199 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.924960 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:13.926429 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.927042 28239 consensus_queue.cc:1225] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:13.927614 28239 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:13.927795 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.928056 28239 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.928480 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 10
W20240426 13:20:13.929487 28265 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.596s	user 0.004s	sys 0.004s
I20240426 13:20:13.929076 28239 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.930580 28239 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.871577 27350 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.897437 28237 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.933279 27350 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.933933 28200 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.934468 28200 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.935952 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 434) took 1546 ms. Trace:
I20240426 13:20:13.936406 25026 rpcz_store.cc:276] 0426 13:20:12.389069 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.389204 (+   135us) service_pool.cc:225] Handling call
0426 13:20:12.424062 (+ 34858us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.666589 (+1242527us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.935661 (+269072us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.935878 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2832465280}
I20240426 13:20:13.922751 24950 rpcz_store.cc:276] 0426 13:20:11.732654 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.567452 (+1834798us) service_pool.cc:225] Handling call
0426 13:20:13.568015 (+   563us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.568479 (+   464us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:13.568819 (+   340us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.724992 (+156173us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.791797 (+ 66805us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.791939 (+   142us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.792205 (+   266us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.792410 (+   205us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.792478 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.922019 (+129541us) raft_consensus.cc:1671] finished
0426 13:20:13.922056 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.922230 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2327,"prepare.queue_time_us":6561,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":1796,"raft.queue_time_us":792,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":10351,"spinlock_wait_cycles":15683200,"thread_start_us":155692,"threads_started":1,"wal-append.queue_time_us":283695}
I20240426 13:20:13.938011 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.938355 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.938620 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.939558 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2094) took 2031 ms. Trace:
I20240426 13:20:13.939628 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.940398 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:13.941419 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.942059 25025 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.943476 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.944591 25029 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:20:13.893110 27943 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:13.909400 27946 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.940058 24952 rpcz_store.cc:276] 0426 13:20:11.908083 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.661794 (+1753711us) service_pool.cc:225] Handling call
0426 13:20:13.696526 (+ 34732us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:13.696953 (+   427us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.696993 (+    40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:13.697595 (+   602us) log.cc:842] Serialized 52 byte log entry
0426 13:20:13.822612 (+125017us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.848644 (+ 26032us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.848967 (+   323us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.849046 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.939359 (+ 90313us) raft_consensus.cc:1671] finished
0426 13:20:13.939382 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.939508 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23904,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":8616,"raft.queue_time_us":6694,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":29863,"thread_start_us":124633,"threads_started":1,"wal-append.queue_time_us":239179}
I20240426 13:20:13.927635 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.938782 25026 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:13.947638 25027 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:13.950896 24950 raft_consensus.cc:2354] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
I20240426 13:20:13.953002 25615 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:13.923827 28254 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.915508 27932 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.958426 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEARNER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:13.961854 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2090) took 2295 ms. Trace:
I20240426 13:20:13.957712 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.966202 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.976547 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:13.979911 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.980644 25104 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:13.981086 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.981282 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.981184 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.982779 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.984252 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.984938 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.987731 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.990489 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.994107 24947 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.994217 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.993822 25026 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:14.004442 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.005877 25023 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:13.982200 25069 connection.cc:232] RPC call timeout handler was delayed by -1.2178523119999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20240426 13:20:14.006057 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.997159 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.007448 25027 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:20:14.020884 25028 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:14.011185 24948 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:14.022089 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64711.9us
I20240426 13:20:14.022630 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:13.990804 28299 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.195s	user 0.001s	sys 0.001s
I20240426 13:20:14.026819 25103 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:13.990546 28248 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:13.984958 28243 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.962404 24954 rpcz_store.cc:276] 0426 13:20:11.666851 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.207002 (+540151us) service_pool.cc:225] Handling call
0426 13:20:12.207373 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.965727 (+758354us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:13.009521 (+ 43794us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.346287 (+336766us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.346355 (+    68us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:13.346412 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.961363 (+614951us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.961424 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.961631 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45312,"prepare.queue_time_us":437686,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":231969,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":15127,"spinlock_wait_cycles":1742523136,"thread_start_us":324782,"threads_started":2,"wal-append.queue_time_us":412886}
I20240426 13:20:14.033341 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.031366 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.036732 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.037823 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.995586 28200 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.041922 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.042409 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.035754 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.046059 24953 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.046530 25623 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.043133 25617 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:14.045354 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.050560 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.051750 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.045800 28141 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:14.057171 28310 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:20:14.061249 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:14.061635 24995 leader_election.cc:401] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.061930 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.063238 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.064160 25026 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:14.065232 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.068588 25022 raft_consensus.cc:2396] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:14.070559 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.071446 25100 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.071493 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.071915 25622 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:14.072324 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.072096 28243 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.072496 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.073491 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.074193 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 582) took 1835 ms. Trace:
I20240426 13:20:14.074302 28243 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.075268 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 584) took 1120 ms. Trace:
I20240426 13:20:14.049942 28254 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.076072 27932 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.077764 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.078455 24952 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:20:14.079681 24952 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.082501 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.080780 27946 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.084717 28280 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.085225 28280 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.085289 25617 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:14.085683 28280 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.086009 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.086869 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.087312 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.087715 28281 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:14.087971 25097 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:20:14.089131 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.094960 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.098182 25024 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:14.075671 24955 rpcz_store.cc:276] 0426 13:20:12.954716 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.970571 (+ 15855us) service_pool.cc:225] Handling call
0426 13:20:14.075120 (+1104549us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.075216 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1259656960}
I20240426 13:20:14.099530 24955 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:14.031818 28248 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:14.100457 28248 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.100719 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:14.101886 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:14.102046 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.103361 28176 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.103793 28176 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.104279 28176 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.074664 24948 rpcz_store.cc:276] 0426 13:20:12.238419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.073390 (+1834971us) service_pool.cc:225] Handling call
0426 13:20:14.073703 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.073889 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.073916 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:14.073958 (+    42us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:14.073987 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.074039 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.074127 (+    88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.102945 28062 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.105942 28317 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:20:14.107023 28062 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.107614 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.108335 24950 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.109015 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.116581 25103 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.117652 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.118847 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.116597 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.089000 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.074549 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.121845 25025 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:14.122925 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.124688 25029 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20240426 13:20:14.112442 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 589) took 1685 ms. Trace:
I20240426 13:20:14.125365 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1737) took 1169 ms. Trace:
I20240426 13:20:14.112696 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1739) took 1156 ms. Trace:
I20240426 13:20:14.125950 24952 rpcz_store.cc:276] 0426 13:20:12.955654 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.079159 (+1123505us) service_pool.cc:225] Handling call
0426 13:20:14.079470 (+   311us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.125032 (+ 45562us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.125289 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.126989 24955 rpcz_store.cc:276] 0426 13:20:12.956616 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.098891 (+1142275us) service_pool.cc:225] Handling call
0426 13:20:14.099300 (+   409us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.099893 (+   593us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.112638 (+ 12745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.078758 25098 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240426 13:20:14.128132 25098 pending_rounds.cc:76] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 5
W20240426 13:20:14.112536 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82661.2us
I20240426 13:20:14.110873 28185 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:14.128876 25098 pending_rounds.cc:97] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240426 13:20:14.109158 28153 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:14.134086 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.124450 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.112455 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.128022 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.110967 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 621) took 3801 ms. Trace:
I20240426 13:20:14.125607 25105 rpcz_store.cc:276] 0426 13:20:12.426695 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.574753 (+1148058us) service_pool.cc:225] Handling call
0426 13:20:13.602510 (+ 27757us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.793631 (+191121us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.112221 (+318590us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54003e17d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.112383 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":432132352}
I20240426 13:20:14.129277 25098 raft_consensus.cc:2866] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:14.139848 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50800.9us
W20240426 13:20:14.139124 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56699.5us
I20240426 13:20:14.142802 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.143944 25097 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:20:14.145367 25097 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:14.148236 28174 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:14.150311 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 586) took 2541 ms. Trace:
I20240426 13:20:14.150774 25098 rpcz_store.cc:276] 0426 13:20:11.608577 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.608761 (+   184us) service_pool.cc:225] Handling call
0426 13:20:11.643232 (+ 34471us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.149261 (+2506029us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.149999 (+   738us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b540009ced4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.150246 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5599939840}
I20240426 13:20:14.158046 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.156185 24950 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:14.159293 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.171689 24954 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:14.199762 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46635.7us
W20240426 13:20:14.201133 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58522.1us
I20240426 13:20:14.155543 24947 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.137877 28281 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:14.196245 28061 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:14.203194 27847 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20240426 13:20:14.203241 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 165436us
I20240426 13:20:14.208679 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:14.212685 28303 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20240426 13:20:14.153649 25621 rpcz_store.cc:276] 0426 13:20:10.327743 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.337603 (+  9860us) service_pool.cc:225] Handling call
0426 13:20:10.413138 (+ 75535us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.413407 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:10.746394 (+332987us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.939775 (+3193381us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b1800078460. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:14.030134 (+ 90359us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:14.036361 (+  6227us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:14.110506 (+ 74145us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.110619 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.110791 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6391,"prepare.queue_time_us":438345,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":60281,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6962,"spinlock_wait_cycles":51846912,"thread_start_us":567882,"threads_started":3,"wal-append.queue_time_us":241973,"wal-append.run_cpu_time_us":1158,"wal-append.run_wall_time_us":1049308}
I20240426 13:20:14.214910 27911 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.216396 28242 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.217823 25618 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:14.218762 28257 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.219717 27677 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:14.220204 28313 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:20:14.196452 27932 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:14.221108 28255 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:14.221177 28268 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.221901 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.221959 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.229918 28281 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.223522 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.224216 28295 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.225454 25029 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:14.226109 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.201799 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.221902 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.254994 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:14.257447 24956 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 13:20:14.258725 24950 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:20:14.261001 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59295.5us
I20240426 13:20:14.261262 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.247289 28242 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.234117 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.162432 28174 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.157951 28025 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.227056 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.244454 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:14.240806 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102189us
W20240426 13:20:14.246416 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51023.4us
I20240426 13:20:14.222786 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.264169 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.268174 25023 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:14.261898 24953 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.262673 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.254741 28281 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.271847 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.269989 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.267316 24947 raft_consensus.cc:2362] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10. Re-sending same reply.
I20240426 13:20:14.267199 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1730) took 2532 ms. Trace:
I20240426 13:20:14.262365 28062 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.286768 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.247280 27677 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.288674 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:14.268893 28295 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.134747 28153 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:14.260625 27932 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.280627 24951 rpcz_store.cc:276] 0426 13:20:11.734418 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.593428 (+1859010us) service_pool.cc:225] Handling call
0426 13:20:13.593974 (+   546us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.660322 (+ 66348us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.660367 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.722962 (+ 62595us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.845537 (+122575us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.854611 (+  9074us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.866316 (+ 11705us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.866535 (+   219us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.266994 (+400459us) raft_consensus.cc:1671] finished
0426 13:20:14.267014 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.267146 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34608,"prepare.queue_time_us":4815,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":73803,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":17905,"spinlock_wait_cycles":3840,"thread_start_us":64452,"threads_started":1,"wal-append.queue_time_us":304661}
I20240426 13:20:14.247344 27911 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.286928 27847 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:14.292161 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.277302 28206 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.302150 28206 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.307736 28206 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.297633 25098 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:14.299149 25105 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.299487 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.295126 28311 raft_consensus.cc:2741] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:20:14.300827 27847 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.310272 25100 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:20:14.290685 28062 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.316372 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.295032 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.264600 28176 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.291369 27350 raft_consensus.cc:2741] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:20:14.328984 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:14.268527 28257 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.299669 28255 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.298478 28061 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:20:14.327097 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:14.332207 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.287987 28248 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.271589 28025 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.337554 28227 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:14.337690 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:14.338018 28227 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.338434 28227 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:14.338567 28248 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:14.339674 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59617.4us
I20240426 13:20:14.337363 28190 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:14.317838 28295 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.322367 28269 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:14.341507 25027 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.338761 28171 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.342319 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.356706 27847 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.358414 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.359549 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.324812 28062 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.351928 28055 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.360702 28055 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.361012 28055 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:14.348153 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121355us
I20240426 13:20:14.362733 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:14.340903 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76880.6us
I20240426 13:20:14.364217 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.365557 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.367316 24950 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20240426 13:20:14.350975 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.368844 25620 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:14.368774 28295 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.369971 28248 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:14.367322 28239 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.353848 27869 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:14.371871 28239 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.371948 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.372488 28239 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.373071 28171 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:14.354876 28305 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:14.355429 27410 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:14.332005 28255 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.330410 28200 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:14.354547 28061 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.367729 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.374583 24953 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.356992 28025 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.348765 28316 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:14.383765 24947 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:20:14.352356 28311 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:20:14.300271 28153 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.390188 24953 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:14.380635 27350 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.381485 28255 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.335832 27932 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.395725 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.383097 28171 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:20:14.382386 28279 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:14.401347 25098 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:14.403272 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.405886 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2097) took 2206 ms. Trace:
I20240426 13:20:14.406320 24949 rpcz_store.cc:276] 0426 13:20:12.199279 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.919623 (+1720344us) service_pool.cc:225] Handling call
0426 13:20:13.943829 (+ 24206us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.994445 (+ 50616us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.994477 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:14.036808 (+ 42331us) log.cc:842] Serialized 26 byte log entry
0426 13:20:14.254277 (+217469us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:14.254334 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.254430 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.405676 (+151246us) raft_consensus.cc:1671] finished
0426 13:20:14.405704 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.405832 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26649,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":24206,"thread_start_us":170738,"threads_started":1,"wal-append.queue_time_us":309378}
I20240426 13:20:14.379926 27869 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240426 13:20:14.407052 28055 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.409451 28061 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.410818 28055 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.412222 27869 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.412936 27869 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.413739 28280 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.414975 24948 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.415486 27869 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.391105 27923 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:14.416293 27923 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.417232 27923 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.418586 27923 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.423686 28300 raft_consensus.cc:2741] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:14.377691 28174 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.426119 27677 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.395323 28200 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20240426 13:20:14.432242 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44388.4us
I20240426 13:20:14.432658 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:14.433509 27869 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:14.434182 27869 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.434409 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.434475 28164 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:14.435544 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.435266 27869 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.436470 28268 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:14.437057 28268 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.438036 28268 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.441155 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.441380 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.389000 28161 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:14.448961 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54053.9us
I20240426 13:20:14.387171 28269 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.443591 24954 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.455600 24949 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:20:14.392172 28025 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:14.462056 25102 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.397197 27350 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:14.450227 28161 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.461970 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.416010 28280 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.457249 27271 raft_consensus.cc:2741] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:14.459986 28257 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.460798 28025 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.467309 27350 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.466382 27946 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:14.470371 28161 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.387537 28242 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.526288 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.534608 28141 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:14.537254 25617 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:14.537629 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.538327 25615 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:20:14.539331 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.539651 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.539964 25619 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:14.537292 25618 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20240426 13:20:14.540740 25618 pending_rounds.cc:76] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 8
I20240426 13:20:14.541024 25618 pending_rounds.cc:97] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20240426 13:20:14.540949 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.541559 25618 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:14.545539 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.508397 28025 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:14.550096 27911 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.551236 27810 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:14.551630 27810 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.482520 28248 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
W20240426 13:20:14.553011 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 73203.3us
I20240426 13:20:14.553704 27410 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.554201 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.554343 27410 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.556618 27946 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:14.555069 28145 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.558051 27946 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.559137 27410 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.558053 28145 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.561092 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.561910 27738 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:14.562296 27738 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.540436 25622 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.563299 27932 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.562312 28145 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.563228 27738 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.565136 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 581) took 2374 ms. Trace:
I20240426 13:20:14.565178 28096 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.565685 27738 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.566253 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1864) took 2111 ms. Trace:
I20240426 13:20:14.565920 28096 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.568567 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.569249 25620 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.569806 28153 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.571708 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 288) took 36905 ms (client timeout 15000 ms). Trace:
W20240426 13:20:14.581540 28307 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.573s	user 0.003s	sys 0.000s
I20240426 13:20:14.575984 28254 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.566597 25104 rpcz_store.cc:276] 0426 13:20:12.454431 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.579535 (+1125104us) service_pool.cc:225] Handling call
0426 13:20:13.579889 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.987361 (+407472us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.566044 (+578683us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400168d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.566202 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921212672}
I20240426 13:20:14.586027 24948 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 10. Ops: []
I20240426 13:20:14.587100 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.587616 24948 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.577628 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:14.588122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69606.3us
I20240426 13:20:14.573288 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 6, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:14.578511 28258 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:14.541519 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.579787 28190 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:14.565582 24956 rpcz_store.cc:276] 0426 13:20:12.191025 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.880141 (+1689116us) service_pool.cc:225] Handling call
0426 13:20:14.077980 (+197839us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:14.257227 (+179247us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:14.259906 (+  2679us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.564953 (+305047us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000785d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.565079 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":414414464}
I20240426 13:20:14.581283 28164 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:14.541893 25624 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.567957 28096 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.597612 25024 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.602205 24954 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:14.469792 27677 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20240426 13:20:14.483803 28237 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20240426 13:20:14.554494 28142 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.476851 28269 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.492254 28206 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.114s
I20240426 13:20:14.620455 24949 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:14.507274 28243 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20240426 13:20:14.627192 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.621748 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.601644 28258 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.628412 25069 leader_election.cc:398] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
I20240426 13:20:14.641197 24949 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:14.641204 25069 leader_election.cc:401] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.643633 24950 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:14.605566 28153 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.643179 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:14.618541 28176 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.620927 27738 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:14.618130 28257 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20240426 13:20:14.676851 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:14.680302 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 308408us
I20240426 13:20:14.681196 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.682291 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.699957 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.701026 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.644306 28034 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20240426 13:20:14.709067 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.649176 28281 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.711493 28284 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:14.712452 28280 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:14.674635 28190 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20240426 13:20:14.715026 28269 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.674607 28239 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.700189 28165 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240426 13:20:14.703359 28164 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.667198 27738 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.717260 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.708946 28227 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.667377 28161 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.717664 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 7, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:14.718770 28171 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.729323 27677 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
W20240426 13:20:14.734127 28323 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:20:14.734759 24956 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.735637 28281 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.736099 28049 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:20:14.708777 28096 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:14.699589 27271 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 13:20:14.581463 24852 rpcz_store.cc:269] 0426 13:19:37.665994 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:37.678006 (+ 12012us) service_pool.cc:225] Handling call
0426 13:20:14.571477 (+36893471us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:05.923809 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:05.975846 (+ 52037us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:05.975901 (+    55us) write_op.cc:427] Acquired schema lock
0426 13:20:05.975921 (+    20us) tablet.cc:660] Decoding operations
0426 13:20:05.978050 (+  2129us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:05.978195 (+   145us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:05.978259 (+    64us) tablet.cc:683] Acquiring locks for 10 operations
0426 13:20:06.016148 (+ 37889us) tablet.cc:699] Row locks acquired
0426 13:20:06.016211 (+    63us) write_op.cc:260] PREPARE: finished
0426 13:20:06.016515 (+   304us) write_op.cc:270] Start()
0426 13:20:06.016780 (+   265us) write_op.cc:275] Timestamp: P: 1714137606016412 usec, L: 0
0426 13:20:06.016840 (+    60us) op_driver.cc:331] REPLICATION: starting
0426 13:20:06.018049 (+  1209us) log.cc:842] Serialized 4020 byte log entry
0426 13:20:08.112524 (+2094475us) op_driver.cc:444] REPLICATION: finished
0426 13:20:08.811709 (+699185us) write_op.cc:300] APPLY: starting
0426 13:20:08.811831 (+   122us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:08.814769 (+  2938us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:08.814902 (+   133us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:08.816311 (+  1409us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:08.816651 (+   340us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0426 13:20:08.816711 (+    60us) write_op.cc:311] APPLY: finished
0426 13:20:08.817070 (+   359us) log.cc:842] Serialized 77 byte log entry
0426 13:20:08.817251 (+   181us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:08.817379 (+   128us) write_op.cc:446] Released schema lock
0426 13:20:08.817444 (+    65us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:08.817806 (+   362us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:08.817835 (+    29us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":446794,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":669790,"apply.run_cpu_time_us":3808,"apply.run_wall_time_us":6481,"num_ops":10,"prepare.queue_time_us":145590,"prepare.run_cpu_time_us":24379,"prepare.run_wall_time_us":375821,"raft.queue_time_us":1180886,"raft.run_cpu_time_us":15166,"raft.run_wall_time_us":469552,"replication_time_us":2090638,"thread_start_us":1501452,"threads_started":4,"wal-append.queue_time_us":464019,"wal-append.run_cpu_time_us":30051,"wal-append.run_wall_time_us":3326021}]]}
I20240426 13:20:14.776651 25027 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:14.780619 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63436.3us
I20240426 13:20:14.721690 28284 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:14.801725 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 446443us
W20240426 13:20:14.781865 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 419172us
I20240426 13:20:14.723984 28280 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.785902 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.743939 27946 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:14.758467 28316 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:14.747946 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 627) took 3665 ms. Trace:
I20240426 13:20:14.783454 28279 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:14.779922 27677 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.783330 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.781829 28096 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.827739 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.783236 28171 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.719065 28062 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.724334 28305 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:14.807184 24853 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:14.806785 28280 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.829034 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:14.815133 28281 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 13:20:14.830418 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44581.6us
I20240426 13:20:14.875577 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.831022 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.831095 28161 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.832141 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.833012 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.876823 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.877019 25098 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.836503 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.836984 27738 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.837066 28284 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.816123 28279 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.838258 28239 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.839143 28227 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.839697 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.907095 25099 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.840364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.841434 27677 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.746434 27810 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.813334 25616 rpcz_store.cc:276] 0426 13:20:11.082553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.428558 (+2346005us) service_pool.cc:225] Handling call
0426 13:20:13.499921 (+ 71363us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.540363 (+ 40442us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.540402 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.541142 (+   740us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.855073 (+313931us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.869506 (+ 14433us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.873580 (+  4074us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.298284 (+424704us) raft_consensus.cc:1671] finished
0426 13:20:14.298305 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.747540 (+449235us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b5400473950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.747768 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113987968,"thread_start_us":288030,"threads_started":1,"wal-append.queue_time_us":526244}
I20240426 13:20:14.827591 27271 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.843881 28025 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.816473 28316 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.804632 27350 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.848155 28096 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.850764 27932 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:14.856518 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.857828 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.858747 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:14.926602 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68850.9us
I20240426 13:20:14.859513 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.860890 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.927608 27738 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.927983 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:14.928284 27738 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.861937 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2101) took 1063 ms. Trace:
I20240426 13:20:14.929231 27738 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.929354 24952 rpcz_store.cc:276] 0426 13:20:13.798926 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.140923 (+341997us) service_pool.cc:225] Handling call
0426 13:20:14.197307 (+ 56384us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:14.197789 (+   482us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.197831 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:14.216153 (+ 18322us) log.cc:842] Serialized 26 byte log entry
0426 13:20:14.453298 (+237145us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:14.453333 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.453389 (+    56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.861725 (+408336us) raft_consensus.cc:1671] finished
0426 13:20:14.861749 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.861887 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":968,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":2443,"thread_start_us":236424,"threads_started":1,"wal-append.queue_time_us":420592}
I20240426 13:20:14.721120 28258 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.931622 28258 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.932963 27677 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.828445 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:20:14.872112 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 325663us
I20240426 13:20:14.873710 25027 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:14.874920 25096 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.937451 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:14.897274 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68315.1us
I20240426 13:20:14.938457 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.902887 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.872383 28171 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.911816 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.913756 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.888607 28281 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.904786 28279 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.915416 24853 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:14.905292 28269 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.924163 28316 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.905193 28295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.020s	sys 0.039s
I20240426 13:20:14.830068 28255 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.904271 28330 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.926434 25022 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.941557 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.941828 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:14.947427 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 300580us
I20240426 13:20:14.952390 24995 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.952870 24995 leader_election.cc:401] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.925573 28284 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.896534 28305 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.957429 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54619.2us
I20240426 13:20:14.960299 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.926050 27932 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.945104 28239 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20240426 13:20:15.039474 25624 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:15.010818 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 386649us
I20240426 13:20:14.943887 28257 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:14.954205 28269 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.969623 28237 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.954825 28206 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
W20240426 13:20:14.958190 28295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.020s	sys 0.040s
W20240426 13:20:14.993327 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54409.9us
W20240426 13:20:15.036319 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 77831.1us
I20240426 13:20:14.945564 27943 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.968250 28243 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:15.010026 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49803.9us
I20240426 13:20:14.954367 28330 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.986785 28325 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:15.009595 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } }
I20240426 13:20:14.986342 28326 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.009737 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.017731 28062 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.944398 28281 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.043174 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.056308 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2108) took 1130 ms. Trace:
I20240426 13:20:15.056564 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.062970 24951 rpcz_store.cc:276] 0426 13:20:13.925713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.306136 (+380423us) service_pool.cc:225] Handling call
0426 13:20:14.306671 (+   535us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:14.307189 (+   518us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.307230 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:14.309099 (+  1869us) log.cc:842] Serialized 52 byte log entry
0426 13:20:14.801718 (+492619us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:14.831022 (+ 29304us) log.cc:842] Serialized 14 byte log entry
0426 13:20:14.831230 (+   208us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.831296 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:15.056098 (+224802us) raft_consensus.cc:1671] finished
0426 13:20:15.056118 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:15.056254 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29541,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":631,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":152,"thread_start_us":413021,"threads_started":1,"wal-append.queue_time_us":673476}
I20240426 13:20:14.986519 27810 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.050798 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.970865 28164 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
W20240426 13:20:15.057817 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57478.5us
I20240426 13:20:15.092015 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:14.989413 28316 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.058477 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:15.056620 28269 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.047545 28200 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.047340 28328 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:15.054658 27943 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.056371 28257 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:15.111850 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61108.7us
W20240426 13:20:15.113037 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69945.8us
I20240426 13:20:15.057996 28055 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:15.058607 28243 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.056589 28325 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.114004 28305 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:15.120111 28255 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.120280 28337 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20240426 13:20:15.056275 28330 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.094189 28062 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:15.064603 28237 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.089428 28152 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.122537 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.126677 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:15.056041 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.064774 28254 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20240426 13:20:15.096653 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.141510 24853 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.114820 28328 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:15.148464 25102 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:15.122404 27943 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.121621 28257 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:15.144580 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.143790 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:15.128795 27738 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.121274 28142 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20240426 13:20:15.144652 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:15.145644 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.170584 25620 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:15.126540 28025 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.130272 27350 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.134655 28243 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.048383 27932 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.175159 27738 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:15.214421 28295 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.230010 27738 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.230657 28280 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.152132 28055 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:15.232528 28055 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.232892 28055 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:15.229394 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98478.6us
I20240426 13:20:15.235080 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.236761 27350 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.141157 28339 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.239601 27932 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:15.243218 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96245.9us
I20240426 13:20:15.244484 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:15.249171 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126690us
I20240426 13:20:15.250274 28268 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:15.250771 28268 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.251258 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.251628 28268 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.251897 28227 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:15.254515 28344 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:15.256328 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:15.257205 27916 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.258325 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.148952 25102 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.117566 28200 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.260149 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.260797 28346 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20240426 13:20:15.261447 28165 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:15.262032 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.263676 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.264058 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.264676 25030 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:15.264959 25025 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:15.265271 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.265472 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.258384 28243 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.266554 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:15.269209 28345 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20240426 13:20:15.212658 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68345.4us
I20240426 13:20:15.237390 27350 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:20:15.278784 28326 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:15.243759 28049 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:15.213438 28343 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20240426 13:20:15.220743 28329 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:20:15.289171 28330 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.364794 27350 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.363940 28049 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.368969 28049 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.369766 28338 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.237s	user 0.002s	sys 0.002s
I20240426 13:20:15.370492 25024 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:15.296281 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 240138us
I20240426 13:20:15.371168 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2112) took 1444 ms. Trace:
I20240426 13:20:15.371469 28330 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.131763 28325 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:15.292446 25096 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:15.297847 25099 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:15.370085 27943 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:15.177155 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75613.4us
I20240426 13:20:15.362920 25022 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:15.375833 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:15.383966 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.151582 28328 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.385342 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.385852 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.386323 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.386618 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.387418 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.388366 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.388505 24853 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.390038 24853 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.377961 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.392330 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.393119 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.393666 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.394145 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.394829 25100 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:15.387428 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.399649 28330 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.143993 28237 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.371523 24953 rpcz_store.cc:276] 0426 13:20:13.926900 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.393260 (+466360us) service_pool.cc:225] Handling call
0426 13:20:14.583354 (+190094us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:14.583554 (+   200us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:14.584014 (+   460us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.584059 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:14.584618 (+   559us) log.cc:842] Serialized 52 byte log entry
0426 13:20:14.811990 (+227372us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:14.812056 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.812122 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:15.251044 (+438922us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:15.370965 (+119921us) raft_consensus.cc:1671] finished
0426 13:20:15.370986 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:15.371122 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17875,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8976,"spinlock_wait_cycles":99176960,"thread_start_us":226995,"threads_started":1,"wal-append.queue_time_us":457700}
I20240426 13:20:15.249328 27738 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:15.383098 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 124832us
I20240426 13:20:15.414503 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:15.144708 28152 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.400894 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.417822 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.353900 25102 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.391276 24853 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:15.387697 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.371654 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.420384 25024 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:15.408627 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:15.151196 25621 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:15.150930 28254 consensus_queue.cc:1225] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:20:15.363896 28326 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.422739 25028 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:15.423466 25025 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:15.423319 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.423729 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.405943 25101 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:15.427482 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.288017 28257 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.381642 28305 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:15.426230 24853 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.440361 25027 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:15.269219 28269 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.410321 27350 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.443926 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.446348 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.450695 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:15.451419 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.451520 25097 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:15.452445 24952 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20240426 13:20:15.452415 25097 raft_consensus.cc:603] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:15.417907 28165 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.426718 28316 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:15.416127 28025 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:15.455698 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.456993 25022 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:15.456298 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:15.463428 25025 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:15.391870 28328 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.464637 24948 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:20:15.467096 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.433768 27810 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.400156 28330 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.440209 28164 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:15.464927 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.479437 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:15.483379 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.488754 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.468600 28034 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 13:20:15.490928 28340 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:15.494055 28340 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.495167 28049 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:15.495064 28340 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.429731 28326 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.497648 28326 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.498025 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.498243 28326 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:15.495616 28049 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:15.499701 25027 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:15.501128 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.499609 28049 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.502144 28152 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.502487 27911 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:15.502745 28152 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.503387 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:15.504135 28049 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:15.503233 27911 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:15.505367 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.512414 24950 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.515959 28339 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240426 13:20:15.520287 28330 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.520504 28339 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:15.520870 28330 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.521077 28339 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.522511 28062 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:15.523058 28062 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.523474 28062 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.448547 27498 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.063s
I20240426 13:20:15.524427 28165 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.520313 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.426486 28171 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.536059 24956 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:15.501168 28096 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.001s
I20240426 13:20:15.478592 27810 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.458745 28061 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 13:20:15.511319 28316 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:15.562590 24947 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:15.569471 27271 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.572590 27943 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.572919 25027 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:20:15.573500 27911 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:15.575232 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.576099 28200 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.471040 27350 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.535559 28164 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.576653 28255 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:15.475407 28025 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.579314 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:15.503283 28152 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.586704 28330 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:15.547556 27810 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.661027 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.664050 25031 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.662638 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.601454 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.665966 24956 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:15.659878 24949 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:15.663735 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.651600 28279 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.679s	user 0.010s	sys 0.001s
I20240426 13:20:15.598544 28190 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20240426 13:20:15.659647 27350 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.572597 28326 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:15.619848 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.619992 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:15.621261 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.585633 28349 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
W20240426 13:20:15.580857 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90351.3us
I20240426 13:20:15.583031 28034 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:15.539520 28280 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:15.651460 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 233685us
I20240426 13:20:15.707432 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.652365 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.616479 28350 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.621150 27847 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20240426 13:20:15.625705 28255 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:15.654495 28328 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.654629 28269 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.655084 28257 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.648725 27738 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:15.655779 28171 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.546609 28096 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.616704 27498 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240426 13:20:15.657104 28166 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20240426 13:20:15.768000 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.718508 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.657984 27923 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.677114 24950 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:15.616017 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:15.651321 28243 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.686774 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.537348 28339 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.584448 28055 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.659677 28152 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.708982 24947 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:15.667034 28237 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.712828 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.770540 24951 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.771266 25620 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20240426 13:20:15.788213 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75450.9us
I20240426 13:20:15.791088 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:15.708483 28284 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.740s	user 0.003s	sys 0.012s
I20240426 13:20:15.724144 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:15.800019 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.801358 25025 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:15.672077 28279 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.714391 28326 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:15.704443 28034 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.660220 28316 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.810308 25624 raft_consensus.cc:2354] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:20:15.815908 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.816797 24951 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:15.802290 28352 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.077s	user 0.001s	sys 0.001s
I20240426 13:20:15.818655 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.813565 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:15.823124 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:15.789677 28049 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.798204 28171 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.845038 25025 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:15.851564 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.851773 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.790143 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.857136 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.857442 25616 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.793649 27847 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.860327 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.860488 27847 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.861167 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.862473 27847 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.816010 28281 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.757s	user 0.010s	sys 0.004s
I20240426 13:20:15.863153 28281 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.865082 28281 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.866334 28243 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.866936 28243 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.806115 28166 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:20:15.856490 28351 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:15.868302 28351 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:15.869027 28096 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.871003 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.871392 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.801738 28190 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240426 13:20:15.826046 28242 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20240426 13:20:15.883065 25100 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:15.869235 28351 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.788192 27923 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.898883 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.899284 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.810480 28305 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20240426 13:20:15.920944 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.921934 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.849673 28295 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.801571 28164 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.942370 25102 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.899979 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.896827 24956 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:15.870098 28339 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.931250 25025 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:15.931483 24947 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:15.931572 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.931281 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.947727 25023 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:15.948664 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.938057 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.936996 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.951228 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.952390 24951 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:15.953671 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:15.964867 25100 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:15.949770 24948 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:15.923035 28340 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:15.972801 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:15.972999 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 244925us
I20240426 13:20:15.979359 25024 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:15.924538 27498 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:15.980875 25024 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 13:20:15.984508 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.985169 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.987339 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.873088 28049 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.947036 28326 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.848453 28255 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.019341 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20240426 13:20:16.012387 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.947072 28025 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:16.021770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 185985us
I20240426 13:20:15.978678 28340 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.982338 27498 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.019908 25025 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 4
I20240426 13:20:16.028170 25025 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20240426 13:20:16.026446 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.989573 27738 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:16.003672 28325 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.631s	user 0.010s	sys 0.001s
I20240426 13:20:16.001704 28161 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.036005 25023 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:20:16.028455 25025 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:16.009272 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.038132 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.038861 24948 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:16.041937 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.035065 24995 leader_election.cc:398] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:20:15.947170 28295 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.044382 24995 leader_election.cc:401] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:20:16.051064 28025 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:16.051409 28062 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s	user 0.006s	sys 0.005s
I20240426 13:20:16.051846 28025 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.052390 28062 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.055955 25101 raft_consensus.cc:595] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.056493 25101 raft_consensus.cc:603] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:16.056977 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.057878 28025 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.059531 25100 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:16.061252 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:16.062516 28025 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:16.064445 28171 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.065091 28171 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.063390 28025 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:16.069841 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.937175 28034 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.036350 28365 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20240426 13:20:16.078006 25023 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:16.034901 27738 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:16.127818 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.080400 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:16.209424 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82344.9us
I20240426 13:20:16.121461 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.211284 24948 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.118000 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.125418 24955 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:20:16.120852 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.213765 24956 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.121735 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.074976 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.178676 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:16.118889 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.218320 24949 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.121157 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.115980 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:16.039893 28242 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.215194 24952 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:16.123795 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.120234 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.229759 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.230146 24951 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:16.229782 25099 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:16.221210 24950 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:16.120536 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.222630 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.179360 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:16.110478 28358 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20240426 13:20:16.109815 28363 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:20:16.245074 24947 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:20:16.067546 28171 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.116946 27923 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.220628 27738 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:16.249013 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.249317 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.251438 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:16.079627 28025 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:16.065238 28062 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.094867 27911 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20240426 13:20:16.109033 28049 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.122601 28295 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.119050 28281 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.296566 28281 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.298962 28281 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.117178 27498 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.300248 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.301424 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.083828 28055 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:16.046111 28161 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.305613 24951 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:16.306515 28161 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.307552 25025 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:16.115264 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:16.311790 25031 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.000s
I20240426 13:20:16.312242 25031 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:16.313333 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.314334 25031 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:16.315181 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.308954 24950 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:16.249083 28164 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20240426 13:20:16.244796 28242 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:16.321686 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 244743us
W20240426 13:20:16.323457 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.796s (SENT)
I20240426 13:20:16.285269 24949 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.324594 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.321728 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:16.327093 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:16.260015 28349 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:20:16.096122 28206 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:16.360819 25026 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:16.040928 28325 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.289724 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.303234 24954 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:16.302033 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:16.302376 25620 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:16.315896 25031 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:16.367976 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.368189 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.368834 25031 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:16.369593 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.128572 28279 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.135538 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:16.195591 28340 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.350277 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 98964.1us
I20240426 13:20:16.371407 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:16.325585 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.372490 25031 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:16.363642 25616 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:16.370939 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 235496us
I20240426 13:20:16.325419 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.364023 25029 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:16.348836 28295 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:16.354568 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 368256us
I20240426 13:20:16.362200 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.168435 28326 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.292220 28380 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20240426 13:20:16.337944 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:16.374783 24950 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:16.300794 27847 raft_consensus.cc:659] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } [suppressed 2 similar messages]
I20240426 13:20:16.361716 28062 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:16.393644 28368 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
I20240426 13:20:16.366123 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.394359 25026 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:16.394613 28361 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.388s	user 0.001s	sys 0.002s
I20240426 13:20:16.398365 28255 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:16.382263 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.400616 25028 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:16.355202 27923 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:16.402226 27923 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:16.402544 25026 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:16.403189 27923 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.380183 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:16.411013 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61099.8us
I20240426 13:20:16.384600 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.412989 25031 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 13:20:16.413260 28380 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:16.413513 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:16.413859 28380 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.413901 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.414693 25031 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.389089 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.391032 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:16.383728 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60308.7us
I20240426 13:20:16.416402 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.400651 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2218) took 1060 ms. Trace:
I20240426 13:20:16.408638 28034 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.419256 25616 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:16.418216 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.442973 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.438618 28344 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:16.384690 28268 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:16.380728 28171 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:16.444428 25029 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:16.444432 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.418570 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.476699 28281 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240426 13:20:16.476861 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58368.5us
I20240426 13:20:16.476994 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.477345 28281 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:16.477926 28281 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.478022 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.452344 27677 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.456323 28340 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:16.483372 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:16.455623 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:16.455889 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 273536us
I20240426 13:20:16.468067 25025 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:16.484905 24947 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:16.467550 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.420259 27932 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:16.452836 28369 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.403s	user 0.000s	sys 0.001s
I20240426 13:20:16.439322 28279 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:16.505980 24955 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 13:20:16.506251 25615 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:16.471871 25102 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:16.385604 28295 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.418198 25022 rpcz_store.cc:276] 0426 13:20:15.339952 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.487730 (+147778us) service_pool.cc:225] Handling call
0426 13:20:15.488255 (+   525us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:15.514400 (+ 26145us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:15.514445 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:15.534522 (+ 20077us) log.cc:842] Serialized 52 byte log entry
0426 13:20:15.888409 (+353887us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:15.888455 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:15.888523 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:16.400381 (+511858us) raft_consensus.cc:1671] finished
0426 13:20:16.400403 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:16.400583 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2354,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":289,"thread_start_us":271216,"threads_started":1,"wal-append.queue_time_us":434465}
W20240426 13:20:16.480242 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 113293us
W20240426 13:20:16.495884 28243 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.626s	user 0.004s	sys 0.008s
I20240426 13:20:16.439383 28381 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:16.482709 24950 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:20:16.468922 28370 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.374s	user 0.001s	sys 0.000s
W20240426 13:20:16.467312 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53867.8us
I20240426 13:20:16.483260 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:16.455740 28239 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:16.466032 27847 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:16.492368 25031 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:20:16.503921 28305 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:16.494149 24947 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:16.466341 28344 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.465812 28062 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.495462 25616 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:16.468155 28254 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.511065 28258 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.500130 24954 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:16.468513 28367 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:16.501279 25025 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:16.474277 28171 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:16.556970 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56996.1us
I20240426 13:20:16.442028 28326 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.551301 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.561985 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:16.503159 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.473690 28375 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:16.503782 24953 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 13:20:16.446135 28055 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:20:16.566668 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.504942 25619 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240426 13:20:16.478119 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.507910 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1900) took 1360 ms. Trace:
I20240426 13:20:16.510059 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.489324 28096 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:16.508224 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.533584 28255 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.572191 25623 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.517580 25022 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:16.518575 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.518208 28200 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20240426 13:20:16.525835 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2233) took 1032 ms. Trace:
I20240426 13:20:16.575539 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:20:16.528213 28387 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20240426 13:20:16.526268 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.499403 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.517161 28243 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.492866 27677 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.502940 27911 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:20:16.510550 28380 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.469406 28268 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.507215 27932 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.568873 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90830.9us
I20240426 13:20:16.568589 25102 rpcz_store.cc:276] 0426 13:20:15.147725 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.147917 (+   192us) service_pool.cc:225] Handling call
0426 13:20:15.148320 (+   403us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:15.778361 (+630041us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b5800a45ab0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:20:15.922527 (+144166us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54001ece20. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902a558d 00007fda902931d0 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59
0426 13:20:16.507592 (+585065us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.507836 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":831315584}
I20240426 13:20:16.520258 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2229) took 1028 ms. Trace:
W20240426 13:20:16.575510 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65532.4us
I20240426 13:20:16.524299 28239 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.588436 28227 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:16.589571 28295 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:16.684919 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 168901us
I20240426 13:20:16.554106 28242 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.554745 28258 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.552800 28367 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:16.591418 24953 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:16.576359 25031 rpcz_store.cc:276] 0426 13:20:15.493096 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.491856 (+998760us) service_pool.cc:225] Handling call
0426 13:20:16.492164 (+   308us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.525567 (+ 33403us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.525784 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:16.694882 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.695514 25031 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:16.695987 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:16.698881 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.593725 24947 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:16.603705 24949 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:16.603899 24955 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.593498 24952 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:20:16.561324 28171 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.621183 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.702785 24951 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:16.590147 27810 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.908s	user 0.006s	sys 0.003s
I20240426 13:20:16.548940 28254 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.616978 25621 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.165s	user 0.013s	sys 0.001s
I20240426 13:20:16.701731 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:16.571933 28279 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:16.704663 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:16.655786 25029 rpcz_store.cc:276] 0426 13:20:15.491978 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.405063 (+913085us) service_pool.cc:225] Handling call
0426 13:20:16.444118 (+ 39055us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:16.476216 (+ 32098us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.520069 (+ 43853us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b540032c8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:16.520200 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88990464}
I20240426 13:20:16.705869 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.600267 28316 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:20:16.608108 28096 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:16.712133 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.702204 28295 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:16.686123 28325 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.592782 28255 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.617909 28375 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.745816 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.750104 24953 raft_consensus.cc:2362] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4. Re-sending same reply.
I20240426 13:20:16.751394 25031 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:16.702534 28367 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:16.621577 28165 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20240426 13:20:16.747658 24955 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.744673 24952 raft_consensus.cc:2362] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4. Re-sending same reply.
W20240426 13:20:16.764994 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60371.2us
I20240426 13:20:16.776389 25588 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:16.592785 28344 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.702239 28242 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
W20240426 13:20:16.757068 28386 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
I20240426 13:20:16.714226 25621 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:16.704267 28381 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.794765 28382 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:16.798626 28366 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:16.802222 27916 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:16.807624 24955 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:16.761449 28279 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:16.758277 28397 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:16.710475 27810 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.818529 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 122609us
I20240426 13:20:16.772182 28295 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:16.830235 28326 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.840492 28295 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:16.840507 28392 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20240426 13:20:16.702275 28396 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20240426 13:20:16.753064 28394 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20240426 13:20:16.844386 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2228) took 1352 ms. Trace:
I20240426 13:20:16.844790 25026 rpcz_store.cc:276] 0426 13:20:15.491723 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.403950 (+912227us) service_pool.cc:225] Handling call
0426 13:20:16.405954 (+  2004us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.407044 (+  1090us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.407085 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.408857 (+  1772us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.530864 (+122007us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:16.530903 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.530968 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:16.844099 (+313131us) raft_consensus.cc:1671] finished
0426 13:20:16.844119 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:16.844273 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2454,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":121175,"threads_started":1,"wal-append.queue_time_us":160657}
I20240426 13:20:16.845806 28394 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:16.839143 28375 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.847057 28402 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:16.847431 28402 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.848562 28402 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.778843 28316 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:16.806878 28384 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20240426 13:20:16.816597 28344 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:16.866577 24955 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:16.846912 28394 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:16.719223 28062 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.785602 28164 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20240426 13:20:16.820425 28366 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:16.839246 28390 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20240426 13:20:16.838335 28391 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.270s	user 0.000s	sys 0.002s
I20240426 13:20:16.892560 25030 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:16.876242 24955 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20240426 13:20:16.838418 28398 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20240426 13:20:16.829618 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 664) took 1759 ms (client timeout 1739 ms). Trace:
I20240426 13:20:16.881153 25026 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:16.779413 28268 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.857039 28316 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:16.925107 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.831287 28397 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.836185 28382 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:16.941752 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 165448us
I20240426 13:20:16.956823 24956 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:16.957180 25029 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:16.880618 28242 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:16.949904 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 251128us
I20240426 13:20:16.952776 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 477) took 1265 ms. Trace:
I20240426 13:20:16.938884 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:16.884299 28243 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.963323 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:16.881322 27677 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.864776 28279 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:16.965833 25618 rpcz_store.cc:276] 0426 13:20:15.687781 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.687894 (+   113us) service_pool.cc:225] Handling call
0426 13:20:15.714717 (+ 26823us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:15.787645 (+ 72928us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:15.787675 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:15.809738 (+ 22063us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.283455 (+473717us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:16.283511 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.283614 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:16.952553 (+668939us) raft_consensus.cc:1671] finished
0426 13:20:16.952585 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:16.952716 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61108,"raft.queue_time_us":73536,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13001,"thread_start_us":412620,"threads_started":1,"wal-append.queue_time_us":521295}
W20240426 13:20:16.959337 28389 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
I20240426 13:20:16.949057 24954 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:16.922576 28402 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:16.847637 28381 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.971645 25104 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:16.977727 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:17.090106 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.968048 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:17.091426 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 123491us
I20240426 13:20:17.092181 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.961935 28258 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.983006 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:16.937690 27932 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.962098 28316 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:17.061151 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:16.992935 25029 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:17.052714 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2224) took 1582 ms. Trace:
I20240426 13:20:17.040115 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 478) took 1015 ms. Trace:
I20240426 13:20:17.050979 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:17.063678 25623 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:17.035620 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80053.7us
W20240426 13:20:17.100275 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49384.3us
W20240426 13:20:17.053197 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 562539us
I20240426 13:20:17.090504 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:16.822571 28096 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.093174 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:17.093575 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 287720us
I20240426 13:20:17.103682 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:17.120314 25105 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:17.038734 28404 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
W20240426 13:20:17.102749 28388 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.012s	sys 0.004s
I20240426 13:20:17.099388 25024 rpcz_store.cc:276] 0426 13:20:15.470149 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.987710 (+517561us) service_pool.cc:225] Handling call
0426 13:20:16.015425 (+ 27715us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.023765 (+  8340us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.023796 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.024258 (+   462us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.320290 (+296032us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:16.320342 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.320431 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.052411 (+731980us) raft_consensus.cc:1671] finished
0426 13:20:17.052440 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.052624 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":704,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":8694,"thread_start_us":244613,"threads_started":1,"wal-append.queue_time_us":349983}
I20240426 13:20:17.109944 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:17.110774 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.982511 28254 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:17.108541 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47487.7us
I20240426 13:20:17.116753 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.118630 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:16.923620 25621 rpcz_store.cc:269] 0426 13:20:15.060346 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.062532 (+  2186us) service_pool.cc:225] Handling call
0426 13:20:16.819341 (+1756809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:17.127838 24954 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:20:17.135166 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2226) took 1644 ms. Trace:
I20240426 13:20:17.099437 25624 rpcz_store.cc:276] 0426 13:20:16.024297 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.024451 (+   154us) service_pool.cc:225] Handling call
0426 13:20:16.024782 (+   331us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.319672 (+294890us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.319703 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.340629 (+ 20926us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.485498 (+144869us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:16.485533 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.485596 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.039801 (+554205us) raft_consensus.cc:1671] finished
0426 13:20:17.039837 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.040043 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26969,"raft.queue_time_us":10532,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":9963,"thread_start_us":389265,"threads_started":2,"wal-append.queue_time_us":159200}
I20240426 13:20:17.179821 25618 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:17.140235 25105 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:17.089478 27677 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:17.142349 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:17.106474 28255 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.104856 28305 raft_consensus.cc:2741] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20240426 13:20:17.110869 28408 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:16.979851 28242 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.031863 28279 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.051550 28374 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20240426 13:20:17.129503 28351 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:17.051288 28061 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20240426 13:20:17.037662 28375 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20240426 13:20:17.117063 28284 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:17.210692 25104 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:17.179275 25027 rpcz_store.cc:276] 0426 13:20:15.491147 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.389859 (+898712us) service_pool.cc:225] Handling call
0426 13:20:16.390196 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.488127 (+ 97931us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:16.488170 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.496750 (+  8580us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.536797 (+ 40047us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:16.536840 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.536904 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.130345 (+593441us) raft_consensus.cc:1671] finished
0426 13:20:17.130373 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.135087 (+  4714us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68025,"raft.queue_time_us":69339,"spinlock_wait_cycles":6625152,"thread_start_us":15103,"threads_started":1,"wal-append.queue_time_us":108522}
I20240426 13:20:17.190802 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:17.213029 25622 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:17.137048 28367 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.135987 28049 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 13:20:17.146351 28409 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.197542 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:17.143697 28395 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:17.214179 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:20:17.178807 28388 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.025s	sys 0.021s
W20240426 13:20:17.192831 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76155.8us
I20240426 13:20:17.051767 28407 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.140s
I20240426 13:20:17.152544 28397 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.238298 25100 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
W20240426 13:20:17.193908 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75335.5us
I20240426 13:20:17.211243 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 607) took 1010 ms. Trace:
I20240426 13:20:17.239905 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.242802 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:17.198307 28165 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.187955 27916 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:17.243681 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:17.245510 25618 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:17.248345 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.197124 27932 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:17.242620 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45140.8us
I20240426 13:20:17.142800 28200 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20240426 13:20:17.197032 28255 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.240751 24948 rpcz_store.cc:276] 0426 13:20:16.200918 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.211966 (+ 11048us) service_pool.cc:225] Handling call
0426 13:20:16.212371 (+   405us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.244352 (+ 31981us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.244389 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.247856 (+  3467us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.450314 (+202458us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:16.450373 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.450466 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.210974 (+760508us) raft_consensus.cc:1671] finished
0426 13:20:17.211002 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.211165 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":519,"raft.queue_time_us":21185,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3644,"thread_start_us":201976,"threads_started":1,"wal-append.queue_time_us":256038}
I20240426 13:20:17.203593 28055 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:17.215651 28284 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:17.208412 28254 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:16.982738 28366 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.252790 28049 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:17.270814 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:17.188659 27677 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.237527 27810 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.256492 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.264909 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.293942 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.278203 28284 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:17.289093 28339 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.336s	user 0.010s	sys 0.004s
I20240426 13:20:17.295622 28254 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.286026 28049 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:17.204186 28408 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.312830 28417 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:17.272583 28142 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:17.315529 28423 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:17.260604 27932 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:17.364497 27932 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:17.364826 28412 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:17.365486 28412 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:17.366746 28412 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:17.367393 28239 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.283604 28055 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:17.182729 28382 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.253281 28410 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:17.325827 28418 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:17.330106 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:17.363636 28171 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.651s	user 0.007s	sys 0.000s
I20240426 13:20:17.353157 25096 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:17.328346 28049 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:17.359308 28430 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
W20240426 13:20:17.362486 28281 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.871s	user 0.008s	sys 0.000s
I20240426 13:20:17.351531 28417 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:17.363705 28427 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.128s	user 0.003s	sys 0.000s
I20240426 13:20:17.260423 28165 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.250949 28409 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.300558 28373 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20240426 13:20:17.272442 28255 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.350337 28351 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:17.433262 28206 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.065s	user 0.011s	sys 0.000s
I20240426 13:20:17.419909 28429 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.428661 28281 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:17.430344 28378 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.065s	user 0.003s	sys 0.000s
I20240426 13:20:17.430274 28432 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:17.428745 28446 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:17.368474 28152 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:17.330121 28061 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.061s
I20240426 13:20:17.448103 28415 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20240426 13:20:17.393982 28239 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:17.432468 28055 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.449429 27677 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.449630 28418 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:17.350055 28164 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.074s
W20240426 13:20:17.363220 27738 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.072s	user 0.004s	sys 0.008s
I20240426 13:20:17.367930 28412 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20240426 13:20:17.423369 28395 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:17.440904 28426 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20240426 13:20:17.353911 28142 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:17.436353 28423 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
W20240426 13:20:17.470641 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 320973us
I20240426 13:20:17.495716 28339 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
W20240426 13:20:17.475210 28025 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.181s	user 0.009s	sys 0.003s
W20240426 13:20:17.510215 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 180184us
W20240426 13:20:17.546761 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 290323us
I20240426 13:20:17.479472 28206 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:20:17.354034 27916 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.434250 28171 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.555267 24951 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:17.538874 28279 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.543390 28242 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.587759 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:17.622224 25622 raft_consensus.cc:2362] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10. Re-sending same reply.
I20240426 13:20:17.623273 27498 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:17.330022 28374 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.137s
W20240426 13:20:17.552635 28413 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20240426 13:20:17.476083 28351 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.550487 28367 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:17.537824 28407 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
W20240426 13:20:17.587793 28416 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
I20240426 13:20:17.618526 28279 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:17.619146 28405 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.358s	user 0.001s	sys 0.000s
W20240426 13:20:17.619608 28436 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20240426 13:20:17.620700 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:17.621260 28171 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.632319 28446 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.629137 27847 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:17.588274 28239 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:17.591953 27738 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:17.679339 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58706.9us
I20240426 13:20:17.614006 28445 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:17.505267 28429 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.594916 28142 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:17.648370 28254 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:17.638880 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 368070us
I20240426 13:20:17.614423 28096 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:17.557660 28316 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.623281 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:17.557121 28418 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:17.660221 25023 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:20:17.628000 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.682271 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.610306 28025 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.699039 25617 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:17.702921 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:17.703339 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:17.681375 28340 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.189s	user 0.006s	sys 0.006s
I20240426 13:20:17.704079 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:17.699378 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2141) took 1161 ms. Trace:
I20240426 13:20:17.701821 28433 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:17.707708 28450 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.708056 28316 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.708408 28410 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:17.708259 28450 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.708941 28418 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:17.710279 28450 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.707230 28437 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.715942 28444 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:17.715775 28437 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.716420 28444 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:17.717290 28444 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.718009 28447 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:17.718397 28447 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:17.718636 28444 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:17.719588 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:17.720310 25624 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:17.680107 28408 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.552325 28417 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.707160 25623 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:17.708585 25621 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:17.690339 28445 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:17.706964 28415 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:17.707121 28281 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:17.721776 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:17.721820 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:17.689092 27847 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.705183 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:17.705171 24952 rpcz_store.cc:276] 0426 13:20:16.537732 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.760195 (+222463us) service_pool.cc:225] Handling call
0426 13:20:16.787273 (+ 27078us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.811376 (+ 24103us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.811430 (+    54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.818358 (+  6928us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.958364 (+140006us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:16.958402 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.958484 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.699125 (+740641us) raft_consensus.cc:1671] finished
0426 13:20:17.699154 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.699304 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":53243,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":84140,"threads_started":1,"wal-append.queue_time_us":240599}
W20240426 13:20:17.737661 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 109773us
I20240426 13:20:17.719272 28447 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.662853 28279 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.596176 28395 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.713284 28374 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.698282 28453 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:17.707710 28433 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:17.713922 28165 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.720329 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2138) took 1199 ms. Trace:
I20240426 13:20:17.746017 28255 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.718111 28437 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:17.684243 28446 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:17.755110 28239 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:17.705369 28096 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:17.899128 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 614) took 1482 ms. Trace:
I20240426 13:20:17.773370 28445 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:17.878208 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:18.045127 24952 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:17.907887 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2137) took 1396 ms. Trace:
I20240426 13:20:17.887823 25099 raft_consensus.cc:595] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:18.039688 24949 rpcz_store.cc:276] 0426 13:20:16.417212 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.417354 (+   142us) service_pool.cc:225] Handling call
0426 13:20:17.898901 (+1481547us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:17.899044 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340625792}
I20240426 13:20:18.048327 25099 raft_consensus.cc:603] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:17.903448 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1914) took 1294 ms. Trace:
I20240426 13:20:17.879616 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2136) took 1370 ms. Trace:
I20240426 13:20:17.774791 28340 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:18.049265 24950 rpcz_store.cc:276] 0426 13:20:16.509308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.573626 (+ 64318us) service_pool.cc:225] Handling call
0426 13:20:16.573962 (+   336us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:16.574347 (+   385us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.574372 (+    25us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:16.616725 (+ 42353us) log.cc:842] Serialized 78 byte log entry
0426 13:20:16.808703 (+191978us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:17.103236 (+294533us) log.cc:842] Serialized 14 byte log entry
0426 13:20:17.570672 (+467436us) log.cc:842] Serialized 14 byte log entry
0426 13:20:17.614496 (+ 43824us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.614574 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.879309 (+264735us) raft_consensus.cc:1671] finished
0426 13:20:17.879338 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.879536 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67769,"prepare.queue_time_us":82660,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"raft.queue_time_us":182121,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":65838,"thread_start_us":819136,"threads_started":3,"wal-append.queue_time_us":233838}
I20240426 13:20:18.047873 24951 rpcz_store.cc:276] 0426 13:20:16.511560 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.703445 (+191885us) service_pool.cc:225] Handling call
0426 13:20:16.718958 (+ 15513us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:17.608981 (+890023us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.907605 (+298624us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:17.907816 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1922891648}
I20240426 13:20:18.061285 25100 rpcz_store.cc:276] 0426 13:20:16.608801 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.611829 (+  3028us) service_pool.cc:225] Handling call
0426 13:20:16.613111 (+  1282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:17.246045 (+632934us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.903199 (+657154us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:17.903387 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1437373184}
I20240426 13:20:17.983785 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 644) took 1497 ms. Trace:
I20240426 13:20:17.893519 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2235) took 1480 ms. Trace:
I20240426 13:20:18.063141 25031 rpcz_store.cc:276] 0426 13:20:16.412926 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.748447 (+335521us) service_pool.cc:225] Handling call
0426 13:20:16.748782 (+   335us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:16.751842 (+  3060us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.893271 (+1141429us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b54002add54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:17.893462 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5035392}
I20240426 13:20:18.048586 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:18.039142 24947 rpcz_store.cc:276] 0426 13:20:16.520439 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.703613 (+183174us) service_pool.cc:225] Handling call
0426 13:20:16.703942 (+   329us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:16.937264 (+233322us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.937304 (+    40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:16.963808 (+ 26504us) log.cc:842] Serialized 52 byte log entry
0426 13:20:17.427688 (+463880us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:17.528638 (+100950us) log.cc:842] Serialized 14 byte log entry
0426 13:20:17.554360 (+ 25722us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.554438 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.720103 (+165665us) raft_consensus.cc:1671] finished
0426 13:20:17.720126 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:17.720277 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27906,"prepare.queue_time_us":100298,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":226483,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":177,"thread_start_us":549515,"threads_started":2,"wal-append.queue_time_us":582027}
W20240426 13:20:17.961549 27932 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.002s
W20240426 13:20:17.901175 28399 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.281s	user 0.003s	sys 0.000s
I20240426 13:20:18.093372 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:18.094172 24949 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:18.094282 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 389138us
W20240426 13:20:17.980787 28454 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20240426 13:20:17.905889 28431 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.622s	user 0.001s	sys 0.001s
W20240426 13:20:18.027911 28402 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.921s	user 0.001s	sys 0.010s
W20240426 13:20:17.908293 28344 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.031s	user 0.007s	sys 0.004s
I20240426 13:20:17.807711 28417 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:18.080945 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.062940 25617 rpcz_store.cc:276] 0426 13:20:16.486484 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.486621 (+   137us) service_pool.cc:225] Handling call
0426 13:20:16.499193 (+ 12572us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:17.704905 (+1205712us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.983473 (+278568us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400380c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:17.983704 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2758862080}
W20240426 13:20:18.078156 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 356426us
I20240426 13:20:18.097885 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.136431 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:18.093573 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:18.013486 28366 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:18.099483 28239 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:18.042038 28034 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.603s	user 0.012s	sys 0.002s
I20240426 13:20:18.104775 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:17.902266 28190 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:17.905097 28279 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.905673 28351 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:17.906805 28429 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:18.107540 24947 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:17.897884 28025 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.098753 28415 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:18.112664 28255 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.043154 28409 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.102723 27932 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.126369 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.040969 27847 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.129786 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.130844 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:18.131700 25102 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20240426 13:20:18.132874 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84361.3us
W20240426 13:20:18.270277 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 139537us
I20240426 13:20:18.270406 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:18.135916 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:17.888523 28447 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:18.078851 28435 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:18.151468 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.299906 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:18.301038 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.162496 24952 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:18.099413 28281 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:17.892273 28455 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:18.050344 28316 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20240426 13:20:18.302825 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 616) took 1710 ms. Trace:
I20240426 13:20:18.301936 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:18.176539 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.116205 28171 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:18.130324 28380 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:18.196758 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 475420us
I20240426 13:20:18.147485 28407 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:17.928512 28049 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.078863 28340 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.145737 28450 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:18.125253 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:18.266433 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2151) took 1157 ms. Trace:
I20240426 13:20:18.323457 24947 rpcz_store.cc:276] 0426 13:20:17.108928 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.106629 (+997701us) service_pool.cc:225] Handling call
0426 13:20:18.107235 (+   606us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.266088 (+158853us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.266361 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:18.324884 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:18.165114 28453 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:17.881747 28374 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.271476 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:18.177857 28423 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.017s	sys 0.060s
I20240426 13:20:18.296411 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:18.333153 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:18.176688 28190 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:18.138957 28437 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20240426 13:20:18.298846 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.150463 28254 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.114s
I20240426 13:20:18.299666 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:18.306386 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2227) took 2815 ms. Trace:
I20240426 13:20:18.337195 28441 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:18.308579 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.337718 28441 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:18.338135 28441 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:18.337430 25028 rpcz_store.cc:276] 0426 13:20:15.491415 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.401297 (+909882us) service_pool.cc:225] Handling call
0426 13:20:16.401609 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.482114 (+ 80505us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.482148 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.505825 (+ 23677us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.537161 (+ 31336us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:16.537197 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.537259 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.900687 (+1363428us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:18.300597 (+399910us) raft_consensus.cc:1671] finished
0426 13:20:18.300617 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.306319 (+  5702us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80139,"raft.queue_time_us":82264,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":7214,"spinlock_wait_cycles":84960128,"thread_start_us":9319,"threads_started":1,"wal-append.queue_time_us":36821}
I20240426 13:20:18.311846 28025 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.340341 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:18.309583 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.340379 28025 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.341365 25031 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:18.342258 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.304682 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2148) took 1197 ms. Trace:
I20240426 13:20:18.342692 28025 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.228008 28279 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:18.313266 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:18.306972 28281 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:18.177482 28239 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.321869 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:18.296411 27738 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:18.306304 28444 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.587s	user 0.007s	sys 0.003s
I20240426 13:20:18.269003 28255 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.307430 24955 rpcz_store.cc:276] 0426 13:20:16.592147 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.914227 (+322080us) service_pool.cc:225] Handling call
0426 13:20:16.914626 (+   399us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.914936 (+   310us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.914971 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.953473 (+ 38502us) log.cc:842] Serialized 26 byte log entry
0426 13:20:17.102834 (+149361us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:17.103119 (+   285us) log.cc:842] Serialized 14 byte log entry
0426 13:20:17.331610 (+228491us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.331693 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:17.905206 (+573513us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:18.297853 (+392647us) raft_consensus.cc:1671] finished
0426 13:20:18.302641 (+  4788us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.302770 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14952,"prepare.queue_time_us":260,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":113212,"spinlock_wait_cycles":31611008,"thread_start_us":267687,"threads_started":2,"wal-append.queue_time_us":230769}
W20240426 13:20:18.336313 28423 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.018s	sys 0.062s
I20240426 13:20:18.342927 24951 rpcz_store.cc:276] 0426 13:20:17.107589 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.078717 (+971128us) service_pool.cc:225] Handling call
0426 13:20:18.113319 (+ 34602us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.113588 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:18.113871 (+   283us) log.cc:842] Serialized 14 byte log entry
0426 13:20:18.235519 (+121648us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:18.235593 (+    74us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:18.303805 (+ 68212us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.304209 (+   404us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.304627 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4942,"prepare.queue_time_us":5453,"prepare.run_cpu_time_us":2939,"prepare.run_wall_time_us":6850,"raft.queue_time_us":33139,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":30468,"thread_start_us":121269,"threads_started":1}
I20240426 13:20:18.264183 28351 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.354282 27810 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:18.346822 28152 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:18.302561 28461 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:18.347681 28325 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:18.379897 25031 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:18.347960 27847 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:18.348271 28254 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:18.348726 28096 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.350216 28340 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:18.172577 28433 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.353088 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 620) took 1586 ms. Trace:
I20240426 13:20:18.318617 28380 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.351894 28049 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:18.411214 28464 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20240426 13:20:18.411513 28049 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:18.411933 28049 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:18.412176 28239 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.412840 28049 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.414232 28049 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:18.414055 28415 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:18.415089 28142 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.701s	user 0.006s	sys 0.000s
I20240426 13:20:18.309558 28171 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:18.355232 28279 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.416261 28171 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:18.379704 28152 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.417599 27932 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.373230 28419 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20240426 13:20:18.418346 28152 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:18.334210 28450 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:18.411237 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66043.5us
I20240426 13:20:18.419204 28255 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.352926 25101 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:18.420351 28450 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:18.420868 28450 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.421355 28450 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.313661 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:18.419840 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:18.423305 25624 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:18.425423 28061 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:18.425976 28061 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:18.426518 25621 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:18.320876 28455 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.427239 28367 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:18.426901 28061 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:18.427367 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72956.7us
I20240426 13:20:18.428272 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 463) took 2390 ms. Trace:
I20240426 13:20:18.428383 28455 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:18.429426 28437 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:18.428629 25022 rpcz_store.cc:276] 0426 13:20:16.037622 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.580810 (+543188us) service_pool.cc:225] Handling call
0426 13:20:16.621013 (+ 40203us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.778701 (+157688us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.778741 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.818750 (+ 40009us) log.cc:842] Serialized 26 byte log entry
0426 13:20:17.088204 (+269454us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:17.088263 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.427895 (+1339632us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:18.427945 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:18.427993 (+    48us) raft_consensus.cc:1671] finished
0426 13:20:18.428006 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.428211 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":574,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":24362,"spinlock_wait_cycles":361656704,"thread_start_us":142585,"threads_started":1,"wal-append.queue_time_us":422091,"wal-append.run_cpu_time_us":1938,"wal-append.run_wall_time_us":1090501}
I20240426 13:20:18.429816 28437 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:18.430548 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 624) took 1178 ms. Trace:
I20240426 13:20:18.431042 24951 rpcz_store.cc:276] 0426 13:20:17.251810 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.360912 (+1109102us) service_pool.cc:225] Handling call
0426 13:20:18.380328 (+ 19416us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.429525 (+ 49197us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:18.429556 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:18.429602 (+    46us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:18.429666 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.430360 (+   694us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54003679d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:18.430400 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.430499 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112682368}
I20240426 13:20:18.352550 28281 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:18.415489 28142 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.433499 28165 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:18.434345 25028 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:18.373454 28351 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.437148 28351 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:18.437832 27498 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.814s	user 0.010s	sys 0.000s
I20240426 13:20:18.438360 28465 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.438855 28465 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.440255 28096 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.440726 28096 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.441201 28344 raft_consensus.cc:979] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:18.441673 28344 raft_consensus.cc:1067] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:18.442435 28096 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.339498 28393 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:18.343605 28452 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.381675 24952 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:18.363035 28423 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:18.405270 24954 rpcz_store.cc:276] 0426 13:20:16.766982 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.182644 (+415662us) service_pool.cc:225] Handling call
0426 13:20:17.201324 (+ 18680us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.385080 (+183756us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:17.385155 (+    75us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:17.395048 (+  9893us) log.cc:842] Serialized 26 byte log entry
0426 13:20:17.629415 (+234367us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:17.629455 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.629517 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:18.352875 (+723358us) raft_consensus.cc:1671] finished
0426 13:20:18.352892 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.353038 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34802,"raft.queue_time_us":128636,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":48445,"thread_start_us":325061,"threads_started":2,"wal-append.queue_time_us":716135}
I20240426 13:20:18.445703 24920 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:18.417680 28279 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.430513 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:18.449029 25022 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:18.442571 28465 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:18.343309 28025 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:18.373096 27810 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:18.416316 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 652) took 1024 ms. Trace:
W20240426 13:20:18.379422 28418 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.670s	user 0.009s	sys 0.000s
I20240426 13:20:18.464812 25618 rpcz_store.cc:276] 0426 13:20:17.391936 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.392151 (+   215us) service_pool.cc:225] Handling call
0426 13:20:17.440994 (+ 48843us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.441389 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:17.441427 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:17.441977 (+   550us) log.cc:842] Serialized 26 byte log entry
0426 13:20:18.013252 (+571275us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:18.013293 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.013369 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:18.416104 (+402735us) raft_consensus.cc:1671] finished
0426 13:20:18.416125 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.416260 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":4400,"thread_start_us":245982,"threads_started":1,"wal-append.queue_time_us":916921}
I20240426 13:20:18.442922 24949 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:18.367578 25103 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
W20240426 13:20:18.378623 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 363572us
I20240426 13:20:18.446055 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:18.443737 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 622) took 1278 ms. Trace:
I20240426 13:20:18.474279 24952 rpcz_store.cc:276] 0426 13:20:17.165754 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.354849 (+1189095us) service_pool.cc:225] Handling call
0426 13:20:18.381240 (+ 26391us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.443497 (+ 62257us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.443674 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:18.326560 28407 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.451686 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:18.473847 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:18.379901 28325 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:18.379379 28461 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:18.496567 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 194689us
I20240426 13:20:18.388216 27847 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:18.467137 24951 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:18.504691 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 625) took 1241 ms. Trace:
I20240426 13:20:18.387645 28254 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:18.316128 28435 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.479282 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:18.482756 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.559s (SENT) [suppressed 25 similar messages]
I20240426 13:20:18.427646 28367 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:18.502343 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:18.512436 25099 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:18.464825 28255 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.464685 28096 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:18.505044 24951 rpcz_store.cc:276] 0426 13:20:17.263146 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.432006 (+1168860us) service_pool.cc:225] Handling call
0426 13:20:18.466949 (+ 34943us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.504401 (+ 37452us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.504640 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:18.533229 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2146) took 1841 ms. Trace:
I20240426 13:20:18.506672 27847 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:18.483570 28284 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.185s	user 0.008s	sys 0.004s
I20240426 13:20:18.332875 28453 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.469242 27810 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:18.472375 27911 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:18.469622 28279 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:18.521836 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2152) took 1129 ms. Trace:
W20240426 13:20:18.543118 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 207817us
I20240426 13:20:18.533710 24956 rpcz_store.cc:276] 0426 13:20:16.691387 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.963567 (+272180us) service_pool.cc:225] Handling call
0426 13:20:16.984253 (+ 20686us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.055356 (+ 71103us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:17.055428 (+    72us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:17.108155 (+ 52727us) log.cc:842] Serialized 26 byte log entry
0426 13:20:17.249598 (+141443us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:17.249637 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.249704 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:18.532985 (+1283281us) raft_consensus.cc:1671] finished
0426 13:20:18.533013 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.533163 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10240,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":20764,"thread_start_us":63454,"threads_started":1,"wal-append.queue_time_us":1188188}
I20240426 13:20:18.503592 28461 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:18.510701 28446 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:18.511854 28254 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:18.524741 28458 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.372s	user 0.001s	sys 0.000s
I20240426 13:20:18.430652 28437 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.508675 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2247) took 1407 ms. Trace:
I20240426 13:20:18.613462 25022 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:18.664342 24949 rpcz_store.cc:276] 0426 13:20:17.392583 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.471135 (+1078552us) service_pool.cc:225] Handling call
0426 13:20:18.471470 (+   335us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.521586 (+ 50116us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.521777 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:18.520942 28450 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:18.451602 28190 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.542249 28316 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:18.656934 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.006s	sys 0.018s
I20240426 13:20:18.461014 28452 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:18.486068 28418 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:18.809177 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 509304us
I20240426 13:20:18.532384 28255 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.808717 25028 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:18.548970 28402 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:18.781677 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 352437us
I20240426 13:20:18.747680 25099 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:18.552834 25621 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240426 13:20:18.915030 25027 rpcz_store.cc:276] 0426 13:20:17.101632 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.714583 (+612951us) service_pool.cc:225] Handling call
0426 13:20:17.722315 (+  7732us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.038330 (+316015us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:18.109909 (+ 71579us) log.cc:842] Serialized 14 byte log entry
0426 13:20:18.507909 (+398000us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:18.507988 (+    79us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:18.508395 (+   407us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.508479 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.508610 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41413,"prepare.queue_time_us":71043,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":50760,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":81,"thread_start_us":389699,"threads_started":2,"wal-append.queue_time_us":373614}
I20240426 13:20:19.055711 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.493124 25105 raft_consensus.cc:2396] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20240426 13:20:18.927969 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:19.494418 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.357s	user 0.011s	sys 0.040s
I20240426 13:20:19.045874 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 650) took 1928 ms. Trace:
W20240426 13:20:19.103575 25589 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.007s	sys 0.039s
W20240426 13:20:19.496377 25589 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.015s	sys 0.063s
W20240426 13:20:19.496711 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.0139e+06us
I20240426 13:20:19.043967 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2210) took 3769 ms. Trace:
I20240426 13:20:19.498533 25622 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:19.499032 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:18.959339 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2242) took 2107 ms. Trace:
W20240426 13:20:19.495054 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.35859e+06us
I20240426 13:20:19.500224 25026 rpcz_store.cc:276] 0426 13:20:16.852344 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.119434 (+267090us) service_pool.cc:225] Handling call
0426 13:20:17.147478 (+ 28044us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:17.325033 (+177555us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:17.325089 (+    56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:17.325144 (+    55us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:17.325213 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.959015 (+1633802us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54001e4454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:18.959106 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:18.959271 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407678720}
I20240426 13:20:19.492949 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:18.938905 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.497706 25030 rpcz_store.cc:276] 0426 13:20:15.274031 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.401880 (+127849us) service_pool.cc:225] Handling call
0426 13:20:15.424257 (+ 22377us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:16.893187 (+1468930us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.043640 (+2150453us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000b0754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:19.043860 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3376196480}
I20240426 13:20:19.495100 25623 rpcz_store.cc:276] 0426 13:20:17.116998 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.117641 (+   643us) service_pool.cc:225] Handling call
0426 13:20:17.117909 (+   268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:17.746637 (+628728us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.045579 (+1298942us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:19.045803 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1354637184}
I20240426 13:20:19.494028 25022 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:19.494161 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:19.182548 28447 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.880s	user 0.008s	sys 0.000s
I20240426 13:20:19.514215 25097 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240426 13:20:19.197188 28492 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:19.522351 28437 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:19.525894 28470 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.859s	user 0.000s	sys 0.000s
I20240426 13:20:19.526623 28380 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:19.527074 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:19.579535 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:19.585005 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1932) took 1074 ms. Trace:
I20240426 13:20:19.585222 25030 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:19.527828 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 623) took 2361 ms. Trace:
W20240426 13:20:19.532814 28254 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.000s
I20240426 13:20:18.664287 28142 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:18.781796 27911 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:18.919144 28467 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.585s	user 0.001s	sys 0.007s
I20240426 13:20:19.596693 24947 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:19.535096 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.007s	sys 0.002s
W20240426 13:20:19.597376 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.007s	sys 0.002s
W20240426 13:20:19.597718 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 702285us
W20240426 13:20:19.598119 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1577) took 3736 ms (client timeout 1687 ms). Trace:
I20240426 13:20:19.598759 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:19.599017 25096 rpcz_store.cc:269] 0426 13:20:15.847121 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.847274 (+   153us) service_pool.cc:225] Handling call
0426 13:20:19.583274 (+3736000us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:19.583394 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3431593216}
I20240426 13:20:18.811390 28469 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:19.535528 25617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.075s	user 0.003s	sys 0.007s
I20240426 13:20:19.536221 28446 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:18.936491 28480 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
W20240426 13:20:19.060927 28484 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.177s	user 0.002s	sys 0.000s
W20240426 13:20:19.082743 25100 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.705s	user 0.005s	sys 0.005s
I20240426 13:20:18.899540 28152 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:19.079010 28445 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.980s	user 0.005s	sys 0.004s
I20240426 13:20:19.537276 28487 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:19.536352 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:18.932334 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 150780us
I20240426 13:20:19.636677 25027 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:19.636967 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:19.637385 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1624) took 1532 ms. Trace:
I20240426 13:20:19.637841 25027 rpcz_store.cc:276] 0426 13:20:18.105079 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.536073 (+1430994us) service_pool.cc:225] Handling call
0426 13:20:19.637313 (+101240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:18.883270 28367 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:18.948623 28482 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.399s	user 0.000s	sys 0.000s
W20240426 13:20:19.537235 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44353.3us
I20240426 13:20:19.656098 28393 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:19.656272 28415 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:19.656693 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:19.656793 28415 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.658066 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:18.937477 28477 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20240426 13:20:19.659256 28415 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.660287 28485 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:19.660459 28415 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:19.660816 28485 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:19.372429 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.207s	user 0.007s	sys 0.011s
W20240426 13:20:19.661432 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.497s	user 0.007s	sys 0.015s
W20240426 13:20:18.954879 28281 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.002s
I20240426 13:20:18.932495 28340 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.691s
I20240426 13:20:18.664921 28325 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:18.784693 28279 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.545562 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2245) took 2686 ms. Trace:
I20240426 13:20:19.579843 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.580145 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.579180 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.585893 24955 rpcz_store.cc:276] 0426 13:20:17.166110 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.357253 (+1191143us) service_pool.cc:225] Handling call
0426 13:20:18.357555 (+   302us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:18.363558 (+  6003us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:18.363589 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:18.387258 (+ 23669us) log.cc:842] Serialized 26 byte log entry
0426 13:20:18.542829 (+155571us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:18.542865 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.542946 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:19.527551 (+984605us) raft_consensus.cc:1671] finished
0426 13:20:19.527579 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:19.527755 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5609,"raft.queue_time_us":53485,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":131626,"threads_started":1,"wal-append.queue_time_us":766437}
I20240426 13:20:19.585403 25104 rpcz_store.cc:276] 0426 13:20:18.510840 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.526812 (+1015972us) service_pool.cc:225] Handling call
0426 13:20:19.584940 (+ 58128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:19.670045 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.669139 25024 rpcz_store.cc:276] 0426 13:20:16.859042 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.359144 (+500102us) service_pool.cc:225] Handling call
0426 13:20:17.394411 (+ 35267us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.394711 (+   300us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:17.394737 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:17.395085 (+   348us) log.cc:842] Serialized 26 byte log entry
0426 13:20:17.655282 (+260197us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:17.655588 (+   306us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:17.655885 (+   297us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:18.922051 (+1266166us) raft_consensus.cc:1671] finished
0426 13:20:18.922073 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:19.545506 (+623433us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34126,"raft.queue_time_us":162305,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":34941,"thread_start_us":178945,"threads_started":1,"wal-append.queue_time_us":608582}
W20240426 13:20:19.158339 28166 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.016s	sys 0.011s
I20240426 13:20:18.908679 28429 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:19.579257 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:20:19.600016 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:19.600719 25617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.140s	user 0.003s	sys 0.008s
I20240426 13:20:19.602402 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:18.951498 28171 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.535s	user 0.003s	sys 0.005s
W20240426 13:20:19.093902 28366 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.929s	user 0.010s	sys 0.000s
I20240426 13:20:18.568262 28284 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.721835 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.722540 25026 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:19.722652 28418 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.610327 25100 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:20:19.607601 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 363389us
I20240426 13:20:19.676877 28281 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:19.621652 24956 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.018s	sys 0.024s
W20240426 13:20:19.725764 24956 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.018s	sys 0.024s
I20240426 13:20:19.661621 28472 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:19.596004 28254 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.550207 28447 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:19.734184 28447 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.607928 28469 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.736207 28447 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:19.736733 28469 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:19.241735 28049 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.827s	user 0.010s	sys 0.002s
W20240426 13:20:19.594574 28500 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20240426 13:20:19.666532 28485 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:18.956863 28453 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.638846 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:19.738123 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 99362.3us
I20240426 13:20:19.738272 28374 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:19.740291 28255 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:19.740762 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1592) took 1610 ms. Trace:
I20240426 13:20:19.741204 28472 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:19.741147 25097 rpcz_store.cc:276] 0426 13:20:18.130719 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.329617 (+198898us) service_pool.cc:225] Handling call
0426 13:20:18.358431 (+ 28814us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:19.514614 (+1156183us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.740522 (+225908us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001b3854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:19.740700 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2657088256}
I20240426 13:20:19.741901 28472 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:19.742414 28472 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:19.742647 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.745077 28496 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:19.687055 28367 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:19.339283 28375 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:19.600524 28501 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20240426 13:20:19.580884 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 680) took 1136 ms. Trace:
W20240426 13:20:19.657407 28499 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.490s	user 0.000s	sys 0.000s
I20240426 13:20:19.548311 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.659230 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:19.104822 27847 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:19.763728 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 104564us
I20240426 13:20:19.765401 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:19.766431 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:19.663882 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.662366 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2248) took 2560 ms. Trace:
I20240426 13:20:19.504640 25105 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
W20240426 13:20:19.421494 28061 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.988s	user 0.000s	sys 0.026s
I20240426 13:20:19.679168 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.768779 25027 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:19.772055 25096 raft_consensus.cc:2396] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:20:19.692827 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.772632 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:18.667907 28423 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.547061 28402 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.621237 28142 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:19.635569 28487 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:19.653276 28445 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.707594 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.620677 28446 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.582334 28492 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.757930 28496 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:19.653717 28410 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:19.656690 28393 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.729319 28171 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.839596 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2153) took 2446 ms. Trace:
W20240426 13:20:19.678663 28166 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.256s	user 0.027s	sys 0.015s
I20240426 13:20:19.726547 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.501996 28096 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:19.625036 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:19.723563 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:19.045536 28441 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.707s	user 0.009s	sys 0.000s
W20240426 13:20:19.694839 28467 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20240426 13:20:19.666631 27677 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:19.863898 27677 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 13:20:19.864215 28461 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.830s	user 0.000s	sys 0.010s
I20240426 13:20:19.864301 27677 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:19.864889 27677 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:19.864826 28461 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.866766 28446 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.661366 28450 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:19.828444 28417 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.664s	user 0.010s	sys 0.001s
I20240426 13:20:19.743827 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.092752 28395 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:19.686589 28455 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:19.726603 28484 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:20:19.751405 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:19.760885 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:19.764665 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.768366 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 621) took 1264 ms. Trace:
I20240426 13:20:19.676883 25022 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:19.767467 25029 rpcz_store.cc:276] 0426 13:20:17.101893 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.842362 (+740469us) service_pool.cc:225] Handling call
0426 13:20:17.884808 (+ 42446us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.924366 (+ 39558us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:17.924413 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:18.013672 (+ 89259us) log.cc:842] Serialized 26 byte log entry
0426 13:20:18.164857 (+151185us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:18.164915 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.165005 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:19.662127 (+1497122us) raft_consensus.cc:1671] finished
0426 13:20:19.662157 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:19.662304 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67926,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":150695,"threads_started":1,"wal-append.queue_time_us":482159}
I20240426 13:20:19.730655 28444 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:19.786327 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 464) took 3666 ms (client timeout 1559 ms). Trace:
I20240426 13:20:19.806458 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2149) took 2698 ms. Trace:
I20240426 13:20:19.547364 28316 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.729396 28483 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:19.813848 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:19.771459 28061 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.338s	user 0.000s	sys 0.030s
I20240426 13:20:19.650045 28152 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.815538 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:19.705556 28429 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.628427 28489 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:19.784781 28049 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:19.830683 28452 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.723501 28284 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.853129 24953 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:19.809090 28326 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:19.808254 28340 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:19.855831 25028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1623) took 1747 ms (client timeout 2133 ms). Trace:
I20240426 13:20:19.852525 25024 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:19.822206 28258 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:19.867069 28461 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:19.860025 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:19.861675 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:19.661145 28415 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:19.820318 28496 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:19.827728 28281 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:19.814877 28142 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.785890 28375 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:20.325301 25028 rpcz_store.cc:269] 0426 13:20:18.104785 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.340086 (+235301us) service_pool.cc:225] Handling call
0426 13:20:19.852207 (+1512121us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b5400415ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:19.852318 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":214461184}
I20240426 13:20:20.364984 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:20.342133 24950 rpcz_store.cc:276] 0426 13:20:17.107928 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.078524 (+970596us) service_pool.cc:225] Handling call
0426 13:20:18.125336 (+ 46812us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.125605 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:18.164706 (+ 39101us) log.cc:842] Serialized 14 byte log entry
0426 13:20:19.803069 (+1638363us) log.cc:842] Serialized 14 byte log entry
0426 13:20:19.805974 (+  2905us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:19.806040 (+    66us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:19.806122 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.806220 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:19.806379 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35825,"prepare.queue_time_us":95221,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":6827,"raft.queue_time_us":60003,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":7836,"thread_start_us":1636457,"threads_started":2,"wal-append.queue_time_us":1521706}
W20240426 13:20:19.992853 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1923) took 1854 ms (client timeout 1545 ms). Trace:
I20240426 13:20:20.326256 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1631) took 1427 ms. Trace:
I20240426 13:20:20.369393 25024 rpcz_store.cc:276] 0426 13:20:18.898340 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.670948 (+772608us) service_pool.cc:225] Handling call
0426 13:20:20.326193 (+655245us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:20.369987 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 512703us
I20240426 13:20:19.869688 25026 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:20.325522 25030 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:20.370923 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1625) took 2265 ms (client timeout 2930 ms). Trace:
W20240426 13:20:20.325037 28506 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.471s	user 0.000s	sys 0.000s
W20240426 13:20:20.371304 25026 rpcz_store.cc:269] 0426 13:20:18.105317 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.723240 (+1617923us) service_pool.cc:225] Handling call
0426 13:20:20.370870 (+647630us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:19.922889 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:20.372439 25100 rpcz_store.cc:269] 0426 13:20:18.072569 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.093310 (+ 20741us) service_pool.cc:225] Handling call
0426 13:20:19.924015 (+1830705us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540009d3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:19.926636 (+  2621us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543001472}
I20240426 13:20:19.810560 28487 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:19.998373 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 326826us
I20240426 13:20:19.770534 25621 rpcz_store.cc:276] 0426 13:20:18.444669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.447666 (+  2997us) service_pool.cc:225] Handling call
0426 13:20:18.496846 (+ 49180us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:19.502313 (+1005467us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.580811 (+ 78498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.340828 25622 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:20:20.341394 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.004s	sys 0.008s
W20240426 13:20:20.403725 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.005s	sys 0.008s
W20240426 13:20:20.404058 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 592820us
W20240426 13:20:20.295933 25623 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.003s	sys 0.013s
W20240426 13:20:20.404871 25623 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.003s	sys 0.014s
I20240426 13:20:20.342252 25102 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:20.281461 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.011s	sys 0.008s
I20240426 13:20:20.343581 25099 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
W20240426 13:20:20.408325 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.012s	sys 0.009s
I20240426 13:20:20.408555 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1598) took 1517 ms. Trace:
I20240426 13:20:20.344012 25105 rpcz_store.cc:276] 0426 13:20:18.504161 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.493924 (+989763us) service_pool.cc:225] Handling call
0426 13:20:19.768314 (+274390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.342103 25096 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
W20240426 13:20:20.344856 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1632) took 1446 ms (client timeout 1867 ms). Trace:
I20240426 13:20:20.327260 25624 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:19.921639 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 13:20:20.326687 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 726666us
W20240426 13:20:20.342558 25023 rpcz_store.cc:269] 0426 13:20:16.097619 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.229514 (+131895us) service_pool.cc:225] Handling call
0426 13:20:19.764087 (+3534573us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003af2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:19.764203 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3185231744}
W20240426 13:20:19.922396 27810 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.138s	user 0.009s	sys 0.000s
I20240426 13:20:20.374472 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:20.375494 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.004s	sys 0.008s
W20240426 13:20:20.373239 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1627) took 2267 ms (client timeout 2147 ms). Trace:
I20240426 13:20:20.385622 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:20.328132 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.389766 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:19.902611 28507 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20240426 13:20:19.921725 28279 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:20.350167 28062 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.001s	sys 0.033s
I20240426 13:20:20.275025 28446 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.409080 25099 rpcz_store.cc:276] 0426 13:20:18.891473 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.764445 (+872972us) service_pool.cc:225] Handling call
0426 13:20:20.408480 (+644035us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:20.491425 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.014s	sys 0.017s
I20240426 13:20:20.461125 25026 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
W20240426 13:20:20.497009 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.014s	sys 0.017s
I20240426 13:20:20.462596 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:20.463491 25031 rpcz_store.cc:269] 0426 13:20:18.105964 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.342025 (+236061us) service_pool.cc:225] Handling call
0426 13:20:20.373067 (+2031042us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b54003f7854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:20.373178 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84920064}
I20240426 13:20:20.464083 25101 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:20.464175 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.007s	sys 0.008s
I20240426 13:20:20.498318 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1629) took 1600 ms. Trace:
I20240426 13:20:20.371879 28521 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:20.428386 25097 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:20.362046 28380 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.468856 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:20.327416 28461 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:20.402824 28503 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.536s	user 0.016s	sys 0.003s
W20240426 13:20:20.482137 28516 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20240426 13:20:20.275302 28453 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:20.471371 25022 rpcz_store.cc:269] 0426 13:20:18.898549 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.504179 (+605630us) service_pool.cc:225] Handling call
0426 13:20:20.344782 (+840603us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:20.486781 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 116226us
I20240426 13:20:20.487880 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.489495 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:20.324903 28340 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.465266 25028 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:20.462069 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:20.488494 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.991s	user 0.000s	sys 0.023s
W20240426 13:20:20.540537 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.043s	user 0.000s	sys 0.024s
I20240426 13:20:20.540603 25030 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:20.541375 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2259) took 1005 ms. Trace:
I20240426 13:20:20.541463 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:20.491569 25619 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.003s	sys 0.019s
I20240426 13:20:20.541720 25030 rpcz_store.cc:276] 0426 13:20:19.538754 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.371057 (+832303us) service_pool.cc:225] Handling call
0426 13:20:20.541322 (+170265us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:20.542295 25619 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.003s	sys 0.019s
I20240426 13:20:20.362299 28444 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:20.464756 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 136711us
I20240426 13:20:20.499960 25026 rpcz_store.cc:276] 0426 13:20:18.897836 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.389424 (+1491588us) service_pool.cc:225] Handling call
0426 13:20:20.498111 (+108687us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.500885 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:20.499867 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 733440us
I20240426 13:20:20.502370 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:20.485489 28062 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.002s	sys 0.034s
I20240426 13:20:19.902827 28441 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.346230 28455 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.527063 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:20.361706 28450 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:19.920228 28096 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.577138 28532 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.577682 28532 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.498589 25024 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:20.579365 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2268) took 1037 ms. Trace:
I20240426 13:20:20.579720 25024 rpcz_store.cc:276] 0426 13:20:19.542309 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.370183 (+827874us) service_pool.cc:225] Handling call
0426 13:20:20.579314 (+209131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.543360 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:19.853528 28410 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.580302 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.522567 28402 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:20.544330 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1628) took 1646 ms. Trace:
I20240426 13:20:20.507972 27847 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:20.508280 28452 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.508576 28445 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:20.566164 25030 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:20.585178 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2251) took 2992 ms. Trace:
I20240426 13:20:20.549394 25031 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:20.585191 25623 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:20.568261 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 104538us
I20240426 13:20:20.532310 28258 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:20.503396 28446 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.362500 28483 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.544324 28461 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.533202 28284 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:20.491528 28254 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:20.616845 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2154) took 3223 ms. Trace:
I20240426 13:20:20.617303 24951 rpcz_store.cc:276] 0426 13:20:17.393218 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.530209 (+1136991us) service_pool.cc:225] Handling call
0426 13:20:18.551862 (+ 21653us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:19.485723 (+933861us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:19.485761 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:19.502912 (+ 17151us) log.cc:842] Serialized 26 byte log entry
0426 13:20:19.728731 (+225819us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:19.728787 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.728860 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:20.616580 (+887720us) raft_consensus.cc:1671] finished
0426 13:20:20.616600 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:20.616784 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":518262,"raft.queue_time_us":1028504,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":217905,"threads_started":1,"wal-append.queue_time_us":356035}
I20240426 13:20:20.581606 25028 rpcz_store.cc:276] 0426 13:20:18.897520 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.374256 (+1476736us) service_pool.cc:225] Handling call
0426 13:20:20.544276 (+170020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.585502 25030 rpcz_store.cc:276] 0426 13:20:17.592679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.563940 (+2971261us) service_pool.cc:225] Handling call
0426 13:20:20.564328 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:20.584949 (+ 20621us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.585128 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.543038 24950 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:20.491233 27810 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:20.591461 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63945.8us
I20240426 13:20:20.591852 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.598870 25031 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:20.629309 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 470) took 2614 ms. Trace:
I20240426 13:20:20.599082 25027 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:20.600651 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 653) took 2287 ms. Trace:
I20240426 13:20:20.629634 25031 rpcz_store.cc:276] 0426 13:20:18.014661 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.598223 (+2583562us) service_pool.cc:225] Handling call
0426 13:20:20.598531 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:20.629071 (+ 30540us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.629260 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.576556 28529 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.590718 28281 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.546054 28492 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:20.630143 25617 rpcz_store.cc:276] 0426 13:20:18.313033 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.313186 (+   153us) service_pool.cc:225] Handling call
0426 13:20:18.330323 (+ 17137us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:18.354761 (+ 24438us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:18.354790 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:18.388441 (+ 33651us) log.cc:842] Serialized 574 byte log entry
0426 13:20:19.698129 (+1309688us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:19.698181 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.698285 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:20.600395 (+902110us) raft_consensus.cc:1671] finished
0426 13:20:20.600424 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:20.600584 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15270,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":10302,"thread_start_us":1237394,"threads_started":1,"wal-append.queue_time_us":1349770}
I20240426 13:20:20.613904 28532 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:20.614105 28495 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
W20240426 13:20:20.614246 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.008s	sys 0.022s
I20240426 13:20:20.546813 28453 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.605137 28445 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:20.671363 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1580) took 4795 ms (client timeout 1892 ms). Trace:
I20240426 13:20:20.672183 25029 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
W20240426 13:20:20.615003 28519 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:20:20.614682 28326 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:20.629653 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.631167 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1621) took 2527 ms. Trace:
I20240426 13:20:20.529688 28380 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:20.607908 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66451.5us
I20240426 13:20:20.643549 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 472) took 2573 ms. Trace:
I20240426 13:20:20.683676 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:20.683235 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53656.2us
I20240426 13:20:20.683198 25031 rpcz_store.cc:276] 0426 13:20:18.104181 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.630420 (+2526239us) service_pool.cc:225] Handling call
0426 13:20:20.630693 (+   273us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:20.630867 (+   174us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:20.630891 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:20.630929 (+    38us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:20.630970 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.631017 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:20.631124 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.685102 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:20.645246 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 106107us
W20240426 13:20:20.648108 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.009s	sys 0.022s
I20240426 13:20:20.686156 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.614568 28520 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.651423 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:20.654417 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 186233us
I20240426 13:20:20.672508 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:20.671921 25103 rpcz_store.cc:269] 0426 13:20:15.848616 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.848748 (+   132us) service_pool.cc:225] Handling call
0426 13:20:20.643521 (+4794773us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:20.643628 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5424907520}
I20240426 13:20:20.678828 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:20.643368 28254 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:20.689664 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:20.498983 28496 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.683457 25030 rpcz_store.cc:276] 0426 13:20:18.070048 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.622778 (+2552730us) service_pool.cc:225] Handling call
0426 13:20:20.643046 (+ 20268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:20.643236 (+   190us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:20.643261 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:20.643303 (+    42us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:20.643341 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.643391 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:20.643501 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.694298 25031 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:20.742914 25100 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
W20240426 13:20:20.685813 28295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.006s	sys 0.003s
I20240426 13:20:20.713699 28446 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:20.711177 28525 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:20.597937 28452 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.712652 28415 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:20.712273 28530 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20240426 13:20:20.711040 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:20.713394 28517 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:20.710026 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:20.714398 28402 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:20.642892 28284 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:20.522027 28521 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.647315 28281 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:21.523522 28525 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:21.514791 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:20.778704 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:20.777786 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:21.524106 25100 pending_rounds.cc:76] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 5
I20240426 13:20:21.244748 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:21.524474 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:20.743781 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:21.521620 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2256) took 1986 ms. Trace:
W20240426 13:20:21.463318 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.006s	sys 0.010s
I20240426 13:20:20.908834 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:20.643765 28529 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:20.778393 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:21.500839 25069 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.030s	sys 0.046s
I20240426 13:20:21.518880 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:20.693387 28492 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:21.571600 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 881926us
I20240426 13:20:21.537282 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:21.584595 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:21.589926 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 894755us
W20240426 13:20:21.590801 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 486994us
I20240426 13:20:20.696326 28254 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:21.273319 28374 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.002s	sys 0.014s
I20240426 13:20:21.597424 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 627) took 4153 ms. Trace:
W20240426 13:20:21.348857 28062 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.008s	sys 0.020s
I20240426 13:20:20.769557 28373 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 13:20:21.537514 28295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.613s	user 0.023s	sys 0.037s
I20240426 13:20:20.742293 28243 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:21.584859 27847 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.016s	sys 0.045s
I20240426 13:20:21.515192 28524 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.765048 28326 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:20.743654 28316 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:21.649510 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2164) took 3280 ms (client timeout 2423 ms). Trace:
I20240426 13:20:21.719731 28453 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:21.515931 28171 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:21.514791 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:20.770957 28152 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:20.682091 28445 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:21.537458 28455 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:21.538193 28340 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:21.519110 28142 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:21.516057 28537 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:20.714819 28520 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:20.768556 28543 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:21.420912 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:21.571910 28457 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.356s
I20240426 13:20:20.770205 28258 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:21.694252 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2255) took 2159 ms (client timeout 1811 ms). Trace:
W20240426 13:20:22.089075 25029 rpcz_store.cc:269] 0426 13:20:19.534887 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.341999 (+807112us) service_pool.cc:225] Handling call
0426 13:20:21.694037 (+1352038us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:21.694167 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":464171136}
I20240426 13:20:21.629645 25030 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:22.090926 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2257) took 2555 ms. Trace:
I20240426 13:20:22.091370 25030 rpcz_store.cc:276] 0426 13:20:19.535519 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:21.570909 (+2035390us) service_pool.cc:225] Handling call
0426 13:20:21.629479 (+ 58570us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.090617 (+461138us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.090862 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:21.957765 24996 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.130s	user 0.023s	sys 0.022s
W20240426 13:20:21.799875 25105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.116s	user 0.006s	sys 0.021s
I20240426 13:20:20.853171 28408 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.809s
I20240426 13:20:21.686972 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:21.516409 28541 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:21.692836 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 684) took 1882 ms. Trace:
I20240426 13:20:21.658983 25026 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:22.002389 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1939) took 1224 ms. Trace:
W20240426 13:20:22.001106 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.992s	user 0.000s	sys 0.039s
I20240426 13:20:20.976673 28096 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:21.686566 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2158) took 3540 ms. Trace:
I20240426 13:20:22.006178 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:22.008116 25031 rpcz_store.cc:276] 0426 13:20:19.535233 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.693629 (+1158396us) service_pool.cc:225] Handling call
0426 13:20:20.693923 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:21.521347 (+827424us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:21.521562 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:20.837160 28429 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:21.589572 28517 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:22.004639 25100 pending_rounds.cc:97] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240426 13:20:22.008783 25103 raft_consensus.cc:2362] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8. Re-sending same reply.
W20240426 13:20:22.009133 25069 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.340s	user 0.038s	sys 0.050s
I20240426 13:20:22.010701 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 483) took 1267 ms. Trace:
W20240426 13:20:22.136864 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1936) took 1367 ms (client timeout 1545 ms). Trace:
I20240426 13:20:22.017089 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1940) took 1238 ms. Trace:
I20240426 13:20:22.136583 25100 raft_consensus.cc:2866] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:22.138005 25096 rpcz_store.cc:276] 0426 13:20:20.778620 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.778838 (+   218us) service_pool.cc:225] Handling call
0426 13:20:22.017028 (+1238190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.134410 24955 rpcz_store.cc:276] 0426 13:20:18.145971 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.669835 (+1523864us) service_pool.cc:225] Handling call
0426 13:20:19.677101 (+  7266us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:19.696193 (+ 19092us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:19.696224 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:19.696781 (+   557us) log.cc:842] Serialized 26 byte log entry
0426 13:20:20.471466 (+774685us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:20.471502 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.471564 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:21.686321 (+1214757us) raft_consensus.cc:1671] finished
0426 13:20:21.686346 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:21.686504 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18688,"raft.queue_time_us":44157,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":54,"thread_start_us":711725,"threads_started":1,"wal-append.queue_time_us":809666}
I20240426 13:20:22.139827 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:22.140373 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:22.013137 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1938) took 1235 ms (client timeout 1578 ms). Trace:
W20240426 13:20:22.137770 25103 rpcz_store.cc:269] 0426 13:20:20.777251 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.777428 (+   177us) service_pool.cc:225] Handling call
0426 13:20:22.136804 (+1359376us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:22.141002 25098 rpcz_store.cc:269] 0426 13:20:20.778028 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.778144 (+   116us) service_pool.cc:225] Handling call
0426 13:20:22.013081 (+1234937us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:22.141508 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.155s	user 0.008s	sys 0.042s
W20240426 13:20:22.141813 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.47255e+06us
I20240426 13:20:22.142249 25070 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
I20240426 13:20:22.142558 25070 leader_election.cc:401] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:22.142796 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1186) took 1363 ms. Trace:
I20240426 13:20:22.011699 24949 rpcz_store.cc:276] 0426 13:20:17.444302 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.919703 (+1475401us) service_pool.cc:225] Handling call
0426 13:20:19.485329 (+565626us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:19.730359 (+245030us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:19.769885 (+ 39526us) log.cc:842] Serialized 14 byte log entry
0426 13:20:20.504628 (+734743us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:20.504716 (+    88us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:20.504823 (+   107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:21.597120 (+1092297us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400078fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:21.597195 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:21.597354 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2895,"prepare.queue_time_us":56003,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":4777,"raft.queue_time_us":112771,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"spinlock_wait_cycles":562926464,"thread_start_us":731441,"threads_started":1,"wal-append.queue_time_us":907383}
I20240426 13:20:22.021791 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:21.990485 28472 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 2.248s	user 0.013s	sys 0.000s
I20240426 13:20:22.009020 24950 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:22.149006 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2173) took 2320 ms (client timeout 2146 ms). Trace:
W20240426 13:20:22.149497 24950 rpcz_store.cc:269] 0426 13:20:19.828898 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.715086 (+886188us) service_pool.cc:225] Handling call
0426 13:20:22.148921 (+1433835us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:21.612201 28497 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.006s	sys 0.012s
W20240426 13:20:21.730876 28367 consensus_meta.cc:287] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.969s	user 0.009s	sys 0.000s
W20240426 13:20:21.690585 28528 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.849s	user 0.000s	sys 0.001s
W20240426 13:20:21.709986 28551 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:20:21.711359 28406 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 4.461s	user 0.013s	sys 0.043s
W20240426 13:20:21.706554 27677 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.841s	user 0.011s	sys 0.000s
I20240426 13:20:21.689363 28200 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:20:22.015259 28062 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.439s	user 0.010s	sys 0.023s
W20240426 13:20:21.727823 28437 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 2.137s	user 0.005s	sys 0.005s
I20240426 13:20:22.015507 28373 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:21.708582 28415 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:21.708335 28418 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:22.015197 28374 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.509s	user 0.004s	sys 0.019s
I20240426 13:20:22.035095 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.160840 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 485) took 1120 ms. Trace:
I20240426 13:20:21.631168 28402 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:21.987727 28450 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:22.022183 28485 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.285s	user 0.011s	sys 0.000s
I20240426 13:20:21.710568 28410 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.003155 28525 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:21.477624 28441 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:22.050184 24947 rpcz_store.cc:269] 0426 13:20:18.319306 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.324598 (+  5292us) service_pool.cc:225] Handling call
0426 13:20:21.599134 (+3274536us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540030c0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:21.599275 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2923081472}
I20240426 13:20:22.015038 28254 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:22.030197 27847 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.264s	user 0.017s	sys 0.045s
I20240426 13:20:22.033056 28550 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.065335 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.038758 28243 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:22.084621 28537 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.085320 28281 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.055615 28326 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.059064 28316 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20240426 13:20:22.086656 28520 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.051138 28524 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:22.092706 24996 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.265s	user 0.023s	sys 0.024s
W20240426 13:20:22.092903 25105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.409s	user 0.017s	sys 0.025s
W20240426 13:20:22.081511 28049 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.735s	user 0.006s	sys 0.006s
I20240426 13:20:22.104794 25623 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:22.077627 28295 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:22.134356 25031 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:22.133603 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 127548us
I20240426 13:20:22.081863 28453 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:22.067576 28452 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:22.009570 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.484s	user 0.014s	sys 0.018s
I20240426 13:20:22.137167 25624 rpcz_store.cc:276] 0426 13:20:20.743358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.743508 (+   150us) service_pool.cc:225] Handling call
0426 13:20:22.010620 (+1267112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.143110 25098 rpcz_store.cc:276] 0426 13:20:20.779833 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.141668 (+1361835us) service_pool.cc:225] Handling call
0426 13:20:22.142745 (+  1077us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.149932 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2260) took 2610 ms. Trace:
I20240426 13:20:22.270881 25030 rpcz_store.cc:276] 0426 13:20:19.539163 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.092427 (+2553264us) service_pool.cc:225] Handling call
0426 13:20:22.149242 (+ 56815us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.149520 (+   278us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:22.149556 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:22.149610 (+    54us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:22.149664 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.149730 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:22.149866 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.152531 25099 rpcz_store.cc:276] 0426 13:20:20.778311 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.778432 (+   121us) service_pool.cc:225] Handling call
0426 13:20:22.002323 (+1223891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.088551 28543 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:22.273630 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.273532 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.274713 25618 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:22.153997 24955 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:22.087410 28549 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.317s	user 0.023s	sys 0.050s
I20240426 13:20:22.158792 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:22.275413 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1786) took 1603 ms (client timeout 1545 ms). Trace:
I20240426 13:20:22.275555 24949 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:22.275749 24955 rpcz_store.cc:269] 0426 13:20:20.672917 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.140151 (+1467234us) service_pool.cc:225] Handling call
0426 13:20:22.275340 (+135189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.158751 25096 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:22.160390 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:22.148417 28455 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 13:20:22.163285 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.028s	sys 0.005s
I20240426 13:20:22.276868 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:22.277276 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.028s	sys 0.005s
I20240426 13:20:22.277361 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1941) took 1498 ms. Trace:
I20240426 13:20:22.278033 25103 rpcz_store.cc:276] 0426 13:20:20.778927 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.141405 (+1362478us) service_pool.cc:225] Handling call
0426 13:20:22.277306 (+135901us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.162123 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:22.155840 28497 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.384s	user 0.022s	sys 0.027s
I20240426 13:20:22.140902 28522 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.294601 28445 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.167639 28561 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
I20240426 13:20:22.296790 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 621) took 5529 ms. Trace:
I20240426 13:20:22.297535 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:22.298051 24949 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:20:22.298566 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1779) took 1659 ms (client timeout 1604 ms). Trace:
W20240426 13:20:22.298916 24949 rpcz_store.cc:269] 0426 13:20:20.639321 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.143821 (+1504500us) service_pool.cc:225] Handling call
0426 13:20:22.298501 (+154680us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:22.299314 25097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1602) took 2755 ms (client timeout 2126 ms). Trace:
I20240426 13:20:22.148525 28517 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:22.300348 28538 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20240426 13:20:22.300909 27810 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.297230 24948 rpcz_store.cc:276] 0426 13:20:16.767744 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:17.272972 (+505228us) service_pool.cc:225] Handling call
0426 13:20:17.273560 (+   588us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:17.541796 (+268236us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:17.564549 (+ 22753us) log.cc:842] Serialized 14 byte log entry
0426 13:20:17.734753 (+170204us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:18.077904 (+343151us) log.cc:842] Serialized 26 byte log entry
0426 13:20:18.078084 (+   180us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:18.078132 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:18.078209 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:22.295430 (+4217221us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b2400274088. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d247ce4 00007fda8d245dfb 00007fda8d2d3e09 00007fda8d2d3a95 00007fda8d2d2ff0 00007fda8ffc62f8 00007fda8ffc6a58 00007fda8ff9e568 00007fda8ff9996a 00007fda8ff98db6
0426 13:20:22.296474 (+  1044us) raft_consensus.cc:1671] finished
0426 13:20:22.296574 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:22.296712 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38012,"prepare.queue_time_us":13155,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":171368,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":11436,"spinlock_wait_cycles":42516736,"thread_start_us":340471,"threads_started":2,"wal-append.queue_time_us":678503}
W20240426 13:20:22.301692 28564 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20240426 13:20:22.301914 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:22.173597 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 150243us
W20240426 13:20:22.170079 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 632833us
I20240426 13:20:22.133165 25619 rpcz_store.cc:276] 0426 13:20:19.810725 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.811368 (+   643us) service_pool.cc:225] Handling call
0426 13:20:19.830797 (+ 19429us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:19.855153 (+ 24356us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:20.542732 (+687579us) log.cc:842] Serialized 14 byte log entry
0426 13:20:20.681937 (+139205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:20.681993 (+    56us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:20.682048 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:21.692620 (+1010572us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:21.692670 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:21.692783 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42587,"prepare.queue_time_us":721147,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":54133,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":18547,"spinlock_wait_cycles":55405952,"thread_start_us":776121,"threads_started":2,"wal-append.queue_time_us":878046}
I20240426 13:20:22.155931 28560 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 12
I20240426 13:20:22.168915 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:22.299710 25097 rpcz_store.cc:269] 0426 13:20:19.543895 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.743572 (+199677us) service_pool.cc:225] Handling call
0426 13:20:22.299113 (+2555541us) spinlock_profiling.cc:243] Waited 87 ms on lock 0x7b54003b06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:22.299244 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200187776}
I20240426 13:20:22.303274 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:22.303432 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 134615us
I20240426 13:20:22.180809 28415 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:21.690294 28375 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.299844 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:22.304080 28415 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:22.304399 28375 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:22.304600 24949 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:22.304827 28375 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:22.305147 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1777) took 1671 ms (client timeout 1753 ms). Trace:
I20240426 13:20:22.163193 28541 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:22.305555 24949 rpcz_store.cc:269] 0426 13:20:20.633964 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.299584 (+1665620us) service_pool.cc:225] Handling call
0426 13:20:22.305078 (+  5494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.127842 28535 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20240426 13:20:22.306638 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:22.307528 24949 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20240426 13:20:22.192493 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:22.307934 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1778) took 1673 ms (client timeout 1890 ms). Trace:
W20240426 13:20:22.308343 24949 rpcz_store.cc:269] 0426 13:20:20.634228 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.306389 (+1672161us) service_pool.cc:225] Handling call
0426 13:20:22.307882 (+  1493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.308570 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:22.172266 28440 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.309243 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:22.202499 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:22.309854 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:22.310114 24949 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20240426 13:20:22.310338 24947 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20240426 13:20:22.310567 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1784) took 1660 ms (client timeout 1889 ms). Trace:
I20240426 13:20:22.310644 25030 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20240426 13:20:22.310776 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1774) took 1677 ms (client timeout 1658 ms). Trace:
W20240426 13:20:22.311236 24947 rpcz_store.cc:269] 0426 13:20:20.633086 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.202259 (+1569173us) service_pool.cc:225] Handling call
0426 13:20:22.310715 (+108456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.312144 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:22.312872 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1781) took 1672 ms (client timeout 1900 ms). Trace:
I20240426 13:20:22.197878 28402 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:22.310911 24949 rpcz_store.cc:269] 0426 13:20:20.652752 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.309000 (+1656248us) service_pool.cc:225] Handling call
0426 13:20:22.310522 (+  1522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.313688 28393 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.314236 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:22.314638 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1930) took 3807 ms (client timeout 2202 ms). Trace:
I20240426 13:20:22.314751 24949 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:22.315212 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2180) took 1769 ms (client timeout 1998 ms). Trace:
W20240426 13:20:22.314996 25101 rpcz_store.cc:269] 0426 13:20:18.507517 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.547208 (+1039691us) service_pool.cc:225] Handling call
0426 13:20:22.314487 (+2767279us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b540001d1d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:22.314578 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1827592064}
I20240426 13:20:22.184404 28200 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:22.316691 28200 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.317727 28537 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.190888 28418 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.200110 28485 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.168676 27677 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.233057 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 461) took 6462 ms. Trace:
I20240426 13:20:22.136472 28284 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:22.234277 28554 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20240426 13:20:22.205046 24956 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.035s	sys 0.046s
I20240426 13:20:22.153690 28096 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.157866 28472 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:22.240155 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.41267e+06us
I20240426 13:20:22.262073 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:19.843192 24952 rpcz_store.cc:276] 0426 13:20:17.392915 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.475193 (+1082278us) service_pool.cc:225] Handling call
0426 13:20:18.475645 (+   452us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:18.476131 (+   486us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:18.509723 (+ 33592us) log.cc:842] Serialized 14 byte log entry
0426 13:20:19.485749 (+976026us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:19.579344 (+ 93595us) log.cc:842] Serialized 52 byte log entry
0426 13:20:19.579507 (+   163us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:19.835918 (+256411us) log.cc:842] Serialized 14 byte log entry
0426 13:20:19.837726 (+  1808us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:19.837812 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:19.837854 (+    42us) raft_consensus.cc:1671] finished
0426 13:20:19.837868 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:19.837986 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":632573,"prepare.queue_time_us":74755,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":13198,"raft.queue_time_us":728556,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":21467,"spinlock_wait_cycles":1844736,"thread_start_us":538997,"threads_started":2,"wal-append.queue_time_us":1088168}
I20240426 13:20:22.268555 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2262) took 2728 ms. Trace:
I20240426 13:20:22.169962 28531 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
W20240426 13:20:22.238984 28553 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20240426 13:20:22.281597 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:22.289625 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1785) took 1631 ms (client timeout 1604 ms). Trace:
I20240426 13:20:22.277493 24955 raft_consensus.cc:2362] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6. Re-sending same reply.
W20240426 13:20:22.270100 28558 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
W20240426 13:20:22.301010 28567 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.002s
I20240426 13:20:22.164912 28552 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.305297 28375 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:22.169793 28373 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:22.315685 24949 rpcz_store.cc:269] 0426 13:20:20.545917 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.314032 (+1768115us) service_pool.cc:225] Handling call
0426 13:20:22.315172 (+  1140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.306237 28541 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.293404 28522 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.239221 28550 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.309334 28440 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:22.313201 24947 rpcz_store.cc:269] 0426 13:20:20.639979 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.311920 (+1671941us) service_pool.cc:225] Handling call
0426 13:20:22.312804 (+   884us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:22.360492 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57598.7us
I20240426 13:20:22.929265 25031 rpcz_store.cc:276] 0426 13:20:19.539803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.133896 (+2594093us) service_pool.cc:225] Handling call
0426 13:20:22.134176 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.268264 (+134088us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.268499 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.359557 25623 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:22.198309 25621 rpcz_store.cc:276] 0426 13:20:21.038948 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:21.200058 (+161110us) service_pool.cc:225] Handling call
0426 13:20:22.159369 (+959311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.931758 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:22.310840 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 484) took 1450 ms. Trace:
W20240426 13:20:22.932201 24950 rpcz_store.cc:269] 0426 13:20:20.664438 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.150368 (+1485930us) service_pool.cc:225] Handling call
0426 13:20:22.289566 (+139198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.933123 25624 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:22.267494 28268 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:22.957650 25031 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:22.958814 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.011s	sys 0.012s
I20240426 13:20:22.939136 25617 rpcz_store.cc:276] 0426 13:20:20.860118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.860271 (+   153us) service_pool.cc:225] Handling call
0426 13:20:21.594014 (+733743us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.014708 (+420694us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:22.014739 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:22.014781 (+    42us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:22.014833 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.310654 (+295821us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:22.310698 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:22.310793 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":967002496}
W20240426 13:20:22.959549 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.011s	sys 0.012s
W20240426 13:20:22.628381 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 324508us
W20240426 13:20:22.933969 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1776) took 2300 ms (client timeout 1727 ms). Trace:
I20240426 13:20:22.267977 28049 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:22.963810 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:22.279852 28549 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.509s	user 0.024s	sys 0.050s
I20240426 13:20:22.964360 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:22.964939 25024 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:22.966032 28441 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.966571 28441 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.274065 28453 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.968883 28441 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:22.413637 27810 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.273633 28429 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.969759 28453 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.969870 28429 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.413352 28535 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.557s
I20240426 13:20:22.972072 28316 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:22.844527 28487 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.378s	user 0.009s	sys 0.005s
I20240426 13:20:22.973886 28034 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.973668 28487 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:22.974493 28415 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.670s	user 0.005s	sys 0.003s
I20240426 13:20:22.974324 28034 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.974929 28415 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.975591 28487 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.842839 28326 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.976087 24951 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:22.976047 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1794) took 1777 ms (client timeout 1642 ms). Trace:
W20240426 13:20:22.976771 24950 rpcz_store.cc:269] 0426 13:20:21.198299 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.975472 (+1777173us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.975952 (+   480us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:22.977138 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1783) took 2334 ms (client timeout 1978 ms). Trace:
I20240426 13:20:22.977180 25622 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9.
W20240426 13:20:22.977687 24949 rpcz_store.cc:269] 0426 13:20:20.642319 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.976948 (+2334629us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.977059 (+   111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:22.976963 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 624) took 4436 ms. Trace:
I20240426 13:20:22.926540 25025 rpcz_store.cc:276] 0426 13:20:15.770272 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.560196 (+789924us) service_pool.cc:225] Handling call
0426 13:20:16.560564 (+   368us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:16.560970 (+   406us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:16.560997 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:16.566208 (+  5211us) log.cc:842] Serialized 26 byte log entry
0426 13:20:16.900651 (+334443us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:16.900710 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:16.900790 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:21.439337 (+4538547us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:22.232709 (+793372us) raft_consensus.cc:1671] finished
0426 13:20:22.232733 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:22.232980 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":993,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9360,"spinlock_wait_cycles":9456512,"thread_start_us":209389,"threads_started":1,"wal-append.queue_time_us":517473}
W20240426 13:20:22.960397 24955 rpcz_store.cc:269] 0426 13:20:20.633686 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.276635 (+1642949us) service_pool.cc:225] Handling call
0426 13:20:22.933908 (+657273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.961793 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.962589 28472 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:22.924427 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 603328us
I20240426 13:20:22.963582 25099 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:22.416684 28452 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:22.959600 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2264) took 3419 ms. Trace:
I20240426 13:20:22.978785 25105 rpcz_store.cc:276] 0426 13:20:18.540802 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.461537 (+1920735us) service_pool.cc:225] Handling call
0426 13:20:20.463990 (+  2453us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:20.655226 (+191236us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:20.655831 (+   605us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:20.668277 (+ 12446us) log.cc:842] Serialized 26 byte log entry
0426 13:20:22.240448 (+1572171us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:22.240486 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.240543 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:22.976768 (+736225us) raft_consensus.cc:1671] finished
0426 13:20:22.976788 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:22.976910 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113050,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":1745096,"threads_started":2,"wal-append.queue_time_us":1491919}
I20240426 13:20:22.962666 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:22.980686 25031 rpcz_store.cc:276] 0426 13:20:19.540414 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.957086 (+3416672us) service_pool.cc:225] Handling call
0426 13:20:22.957443 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.959338 (+  1895us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.959551 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:22.981256 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:22.923193 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 601176us
I20240426 13:20:22.969522 27810 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:22.971812 28429 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:22.979166 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:22.360906 24948 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20240426 13:20:22.979950 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:22.982065 25022 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:22.978754 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1782) took 2336 ms (client timeout 2146 ms). Trace:
I20240426 13:20:22.982721 24954 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:22.972568 28316 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.982803 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:22.980839 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:22.978951 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1790) took 2015 ms (client timeout 1825 ms). Trace:
I20240426 13:20:22.971343 28550 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:22.280375 28455 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:22.985205 25097 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:20:23.053117 28407 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:23.059528 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80437.2us
W20240426 13:20:23.060391 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1775) took 2427 ms (client timeout 1786 ms). Trace:
I20240426 13:20:22.667174 28295 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:23.057363 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:23.064348 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:20:23.056406 24952 rpcz_store.cc:269] 0426 13:20:20.641985 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.978527 (+2336542us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.978633 (+   106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:22.797485 28444 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:22.984387 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1792) took 2020 ms (client timeout 1893 ms). Trace:
W20240426 13:20:23.058158 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1780) took 2338 ms (client timeout 2212 ms). Trace:
W20240426 13:20:23.036823 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1801) took 1654 ms (client timeout 1578 ms). Trace:
W20240426 13:20:23.058329 24947 rpcz_store.cc:269] 0426 13:20:20.963509 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.978799 (+2015290us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.978895 (+    96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:23.058763 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77998.9us
W20240426 13:20:23.067919 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1800) took 1738 ms (client timeout 1626 ms). Trace:
W20240426 13:20:23.068094 24955 rpcz_store.cc:269] 0426 13:20:21.330301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.983947 (+1653646us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.984078 (+   131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.068298 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:22.984691 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:23.018087 25023 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20240426 13:20:23.060818 24948 rpcz_store.cc:269] 0426 13:20:20.633400 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.301668 (+1668268us) service_pool.cc:225] Handling call
0426 13:20:23.060303 (+758635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:23.053532 27677 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:23.065668 24949 rpcz_store.cc:269] 0426 13:20:20.964102 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.984167 (+2020065us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.984324 (+   157us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:22.359570 28529 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.666265 28560 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
I20240426 13:20:23.063586 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:22.798195 28524 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:22.278451 28543 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:23.066174 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:23.068367 24952 rpcz_store.cc:269] 0426 13:20:21.329283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.067763 (+1738480us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.067862 (+    99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.035990 28472 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240426 13:20:23.070637 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:23.051611 28455 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:23.072474 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1789) took 2321 ms (client timeout 2209 ms). Trace:
W20240426 13:20:23.089709 24947 rpcz_store.cc:269] 0426 13:20:20.963094 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.067974 (+2104880us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.068233 (+   259us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:22.983556 28441 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:23.072734 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1795) took 1870 ms (client timeout 1787 ms). Trace:
W20240426 13:20:23.091390 24955 rpcz_store.cc:269] 0426 13:20:21.198611 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.069052 (+1870441us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.069123 (+    71us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.073465 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:23.074366 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1787) took 2396 ms (client timeout 2329 ms). Trace:
I20240426 13:20:23.062085 28531 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:23.062283 28049 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:22.958863 28552 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:22.527786 28243 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:23.076159 28444 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:23.016705 28452 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:23.081811 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:23.122253 24952 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:23.067570 24950 rpcz_store.cc:269] 0426 13:20:20.639665 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.977601 (+2337936us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:22.977699 (+    98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.085652 28418 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.086210 28435 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:23.087668 28521 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.081015 25101 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:23.078512 28096 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:23.039316 27916 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
W20240426 13:20:23.073153 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1804) took 1740 ms (client timeout 1721 ms). Trace:
W20240426 13:20:23.134830 24949 rpcz_store.cc:269] 0426 13:20:21.344355 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.073009 (+1728654us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.073110 (+   101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:23.092603 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1799) took 1764 ms (client timeout 1761 ms). Trace:
W20240426 13:20:23.092816 24948 rpcz_store.cc:269] 0426 13:20:20.678402 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.074193 (+2395791us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.074308 (+   115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.088721 25103 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
W20240426 13:20:23.096474 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1802) took 1765 ms (client timeout 1759 ms). Trace:
I20240426 13:20:23.075371 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:23.137292 24947 rpcz_store.cc:269] 0426 13:20:21.331238 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.096250 (+1765012us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.096394 (+   144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.137579 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:23.111841 28049 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:23.112704 28444 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:23.115321 28552 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:23.116165 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:23.118978 28295 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:23.098357 28531 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:23.130468 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 203645us
I20240426 13:20:23.131125 28360 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:23.132787 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:23.134410 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:23.136219 24955 rpcz_store.cc:269] 0426 13:20:21.328309 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.092374 (+1764065us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.092527 (+   153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:23.127612 28407 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:23.118808 28373 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:23.132681 28375 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:23.139211 24948 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:23.146962 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:23.142616 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:23.244813 24956 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.895s	user 0.043s	sys 0.058s
W20240426 13:20:23.769197 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1793) took 2571 ms (client timeout 1979 ms). Trace:
W20240426 13:20:23.768906 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1797) took 2569 ms (client timeout 1989 ms). Trace:
I20240426 13:20:23.749768 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:23.150864 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:23.107960 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2263) took 3567 ms. Trace:
I20240426 13:20:23.148934 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:23.139878 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:23.134554 28415 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.257551 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2250) took 5665 ms. Trace:
W20240426 13:20:23.825989 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.001s	sys 0.008s
I20240426 13:20:23.961583 28373 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:23.962142 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.001s	sys 0.009s
W20240426 13:20:23.161223 28555 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 1.571s	user 0.006s	sys 0.017s
I20240426 13:20:23.962445 28375 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:23.156029 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2266) took 3615 ms (client timeout 1589 ms). Trace:
I20240426 13:20:23.154175 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 471) took 5100 ms. Trace:
W20240426 13:20:23.827116 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 694191us
I20240426 13:20:23.098351 28441 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:23.133198 28524 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:23.770306 24948 rpcz_store.cc:269] 0426 13:20:21.207227 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.137225 (+1929998us) service_pool.cc:225] Handling call
0426 13:20:23.768668 (+631443us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:23.251170 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.003s	sys 0.006s
W20240426 13:20:23.592079 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.023s	sys 0.032s
W20240426 13:20:23.769944 24950 rpcz_store.cc:269] 0426 13:20:21.197951 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.130558 (+1932607us) service_pool.cc:225] Handling call
0426 13:20:23.768963 (+638405us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:23.697515 28243 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.010s	sys 0.005s
I20240426 13:20:23.152580 28498 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
W20240426 13:20:23.670083 28541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.000s	sys 0.016s
I20240426 13:20:23.155961 28316 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:23.947197 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.961s	user 0.004s	sys 0.010s
W20240426 13:20:24.001827 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.005s	sys 0.010s
I20240426 13:20:23.974957 28481 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:24.002271 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.04102e+06us
I20240426 13:20:23.946955 28407 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:23.650173 28382 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.004s	sys 0.012s
W20240426 13:20:23.669221 28580 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.003s	sys 0.013s
W20240426 13:20:24.003032 28541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s	user 0.001s	sys 0.018s
W20240426 13:20:23.144172 28254 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.860s	user 0.008s	sys 0.001s
I20240426 13:20:23.153762 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2253) took 5513 ms. Trace:
W20240426 13:20:23.533932 28402 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.220s	user 0.010s	sys 0.001s
I20240426 13:20:23.546113 27810 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.301235 28585 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:23.562844 28452 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.955647 25098 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:23.957739 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:23.161828 28401 consensus_queue.cc:1159] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 9 status { last_received { term: 9 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 9 index: 5 } }
I20240426 13:20:23.156082 27916 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
W20240426 13:20:23.154172 28517 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.854s	user 0.007s	sys 0.002s
I20240426 13:20:23.951231 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:20:23.983744 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2177) took 4116 ms (client timeout 1895 ms). Trace:
W20240426 13:20:23.963840 24952 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.837s	user 0.006s	sys 0.001s
W20240426 13:20:23.230123 28525 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.018s	user 0.011s	sys 0.000s
I20240426 13:20:23.911553 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:23.146489 28444 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:23.964730 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.009s	user 0.042s	sys 0.049s
I20240426 13:20:23.162446 28531 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:23.155118 28485 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:23.963795 25027 rpcz_store.cc:269] 0426 13:20:19.540986 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.772418 (+231432us) service_pool.cc:225] Handling call
0426 13:20:23.155826 (+3383408us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:23.155944 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1898866944}
I20240426 13:20:23.976958 25097 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20240426 13:20:23.961197 25022 rpcz_store.cc:276] 0426 13:20:17.592326 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.545559 (+2953233us) service_pool.cc:225] Handling call
0426 13:20:20.576748 (+ 31189us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:23.057530 (+2480782us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:23.257270 (+199740us) spinlock_profiling.cc:243] Waited 2.4 s on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:23.257479 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5520569088}
W20240426 13:20:23.980763 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.001s	sys 0.010s
I20240426 13:20:23.972295 25028 rpcz_store.cc:276] 0426 13:20:18.054211 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.621246 (+2567035us) service_pool.cc:225] Handling call
0426 13:20:20.621660 (+   414us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:22.215720 (+1594060us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b18004a72e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda903100fb 00007fda9031a91b
0426 13:20:22.233581 (+ 17861us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:22.233607 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:22.234039 (+   432us) log.cc:842] Serialized 26 byte log entry
0426 13:20:22.960446 (+726407us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:22.960483 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:22.960557 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:23.153878 (+193321us) raft_consensus.cc:1671] finished
0426 13:20:23.153898 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:23.154115 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13534,"raft.queue_time_us":12869,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10560,"spinlock_wait_cycles":9947392,"thread_start_us":776907,"threads_started":2,"wal-append.queue_time_us":855937}
W20240426 13:20:23.981307 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.347s	user 0.003s	sys 0.009s
I20240426 13:20:23.981559 28441 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:23.977783 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1798) took 2685 ms (client timeout 2249 ms). Trace:
W20240426 13:20:23.981818 28243 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.010s	sys 0.006s
I20240426 13:20:23.982192 28498 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:23.982574 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.025s	sys 0.035s
W20240426 13:20:24.031585 24955 rpcz_store.cc:269] 0426 13:20:21.285663 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.150618 (+1864955us) service_pool.cc:225] Handling call
0426 13:20:23.968937 (+818319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.040887 28498 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.042162 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:24.043229 24955 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:23.951251 28360 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.044399 28517 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.046517 28360 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:23.983688 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1805) took 2528 ms (client timeout 2177 ms). Trace:
I20240426 13:20:23.965319 25030 rpcz_store.cc:276] 0426 13:20:19.540129 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.311220 (+2771091us) service_pool.cc:225] Handling call
0426 13:20:22.319714 (+  8494us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.319899 (+   185us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:22.926624 (+606725us) log.cc:842] Serialized 14 byte log entry
0426 13:20:23.106863 (+180239us) log.cc:842] Serialized 14 byte log entry
0426 13:20:23.107333 (+   470us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:23.107396 (+    63us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:23.107467 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:23.107558 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:23.107895 (+   337us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":758573,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":246,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":201,"thread_start_us":625936,"threads_started":2,"wal-append.queue_time_us":179474}
I20240426 13:20:23.984608 28483 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:23.985870 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:24.048765 25616 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:24.048691 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:24.048765 25623 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:20:23.994130 28524 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.049794 25030 raft_consensus.cc:2362] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4. Re-sending same reply.
I20240426 13:20:23.989567 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2265) took 4448 ms. Trace:
I20240426 13:20:23.154976 28049 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:23.995216 24996 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.034s	sys 0.026s
I20240426 13:20:23.998823 28034 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.005013 28481 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.005311 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2252) took 6365 ms. Trace:
I20240426 13:20:24.071300 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2249) took 6960 ms. Trace:
I20240426 13:20:24.014487 25024 rpcz_store.cc:276] 0426 13:20:17.640623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.580450 (+2939827us) service_pool.cc:225] Handling call
0426 13:20:20.612780 (+ 32330us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.965337 (+2352557us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:23.153507 (+188170us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:23.153706 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5408989056}
W20240426 13:20:24.017733 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 490) took 2359 ms (client timeout 2249 ms). Trace:
I20240426 13:20:24.017834 25101 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:24.019052 28380 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.019491 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1946) took 1085 ms. Trace:
I20240426 13:20:24.026070 27916 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.020648 27911 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.022092 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 628) took 1064 ms. Trace:
I20240426 13:20:24.022559 28522 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.023638 28461 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:24.005630 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2181) took 1948 ms (client timeout 1750 ms). Trace:
W20240426 13:20:24.026038 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1796) took 2827 ms (client timeout 2063 ms). Trace:
W20240426 13:20:24.013468 28580 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.893s	user 0.004s	sys 0.015s
W20240426 13:20:24.099910 24950 rpcz_store.cc:269] 0426 13:20:22.058532 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.005387 (+1946855us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:24.005549 (+   162us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:24.101437 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1949) took 1160 ms. Trace:
W20240426 13:20:24.003527 28543 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.907s	user 0.011s	sys 0.007s
W20240426 13:20:24.028796 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1791) took 3065 ms (client timeout 2225 ms). Trace:
W20240426 13:20:24.014987 28382 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.053s	user 0.005s	sys 0.014s
I20240426 13:20:24.101871 25101 rpcz_store.cc:276] 0426 13:20:22.939580 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.957016 (+1017436us) service_pool.cc:225] Handling call
0426 13:20:24.095130 (+138114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.049327 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63740us
I20240426 13:20:23.995105 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1945) took 1061 ms. Trace:
W20240426 13:20:24.052160 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.118s	user 0.042s	sys 0.050s
W20240426 13:20:24.047307 24948 rpcz_store.cc:269] 0426 13:20:21.454928 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.983379 (+2528451us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:23.983606 (+   227us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:24.014382 28407 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:24.015614 28419 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:24.056661 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:24.051476 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.002s	sys 0.010s
I20240426 13:20:24.057201 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2258) took 4521 ms. Trace:
I20240426 13:20:24.071826 25023 rpcz_store.cc:276] 0426 13:20:17.110910 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.529852 (+3418942us) service_pool.cc:225] Handling call
0426 13:20:20.530236 (+   384us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:23.069555 (+2539319us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.070683 (+1001128us) spinlock_profiling.cc:243] Waited 2.49 s on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.071101 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5720778112}
I20240426 13:20:24.059720 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.060653 28596 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:24.063566 28360 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:24.110910 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.84874e+06us
I20240426 13:20:24.078547 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:24.025081 24951 rpcz_store.cc:269] 0426 13:20:19.868952 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.692297 (+823345us) service_pool.cc:225] Handling call
0426 13:20:23.983253 (+3290956us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:23.983536 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2239371648}
W20240426 13:20:24.100198 24949 rpcz_store.cc:269] 0426 13:20:21.198902 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.138123 (+1939221us) service_pool.cc:225] Handling call
0426 13:20:24.025916 (+887793us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.077885 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:24.113802 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.114111 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:24.114807 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1807) took 1188 ms. Trace:
I20240426 13:20:24.114742 28543 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.115475 24949 rpcz_store.cc:276] 0426 13:20:22.926351 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.113359 (+1187008us) service_pool.cc:225] Handling call
0426 13:20:24.114724 (+  1365us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.096781 25098 rpcz_store.cc:276] 0426 13:20:22.934050 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.070391 (+136341us) service_pool.cc:225] Handling call
0426 13:20:24.019258 (+948867us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.019436 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1958826752}
I20240426 13:20:24.015851 28585 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:24.095233 25622 rpcz_store.cc:269] 0426 13:20:21.657625 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.021505 (+363880us) service_pool.cc:225] Handling call
0426 13:20:24.017151 (+1995646us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.017406 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1915110144}
W20240426 13:20:24.098664 24996 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.034s	sys 0.026s
I20240426 13:20:24.047535 28517 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:24.104456 24947 rpcz_store.cc:269] 0426 13:20:20.963827 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.137917 (+2174090us) service_pool.cc:225] Handling call
0426 13:20:24.026180 (+888263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.106755 25103 rpcz_store.cc:276] 0426 13:20:22.933788 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.963562 (+ 29774us) service_pool.cc:225] Handling call
0426 13:20:23.994668 (+1031106us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:23.994874 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259045120}
I20240426 13:20:24.108369 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.112172 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.068192 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1951) took 1099 ms. Trace:
W20240426 13:20:24.074579 25589 connection.cc:232] RPC call timeout handler was delayed by -3.0686126760000434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
I20240426 13:20:24.071615 25026 rpcz_store.cc:276] 0426 13:20:17.640287 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.550341 (+2910054us) service_pool.cc:225] Handling call
0426 13:20:20.585553 (+ 35212us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.153142 (+1567589us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.004683 (+1851541us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.005102 (+   419us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2468218240}
I20240426 13:20:24.115984 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:24.051458 25031 rpcz_store.cc:276] 0426 13:20:19.540696 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.981686 (+3440990us) service_pool.cc:225] Handling call
0426 13:20:22.981949 (+   263us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:22.983546 (+  1597us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:22.983575 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:23.016610 (+ 33035us) log.cc:842] Serialized 26 byte log entry
0426 13:20:23.963071 (+946461us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:23.963205 (+   134us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:23.963372 (+   167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:23.989090 (+ 25718us) raft_consensus.cc:1671] finished
0426 13:20:23.989135 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:23.989438 (+   303us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1261,"raft.queue_time_us":2297,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":19277,"thread_start_us":900730,"threads_started":1,"wal-append.queue_time_us":946442}
W20240426 13:20:24.120991 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.04747e+06us
I20240426 13:20:24.128232 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.117337 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:24.118142 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:24.098881 25105 rpcz_store.cc:276] 0426 13:20:22.957620 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.982586 (+ 24966us) service_pool.cc:225] Handling call
0426 13:20:24.021718 (+1039132us) spinlock_profiling.cc:243] Waited 64.4 ms on lock 0x7b54003ef8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.021909 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148184448}
I20240426 13:20:24.124223 25097 rpcz_store.cc:276] 0426 13:20:22.968387 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.137792 (+169405us) service_pool.cc:225] Handling call
0426 13:20:24.068072 (+930280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.081686 28524 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.119009 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2287) took 1061 ms. Trace:
I20240426 13:20:24.095341 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:24.121003 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.074998 28592 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:24.135545 25589 connection.cc:232] RPC call timeout handler was delayed by -2.8526126760000352s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1774s for a response
I20240426 13:20:24.117857 28543 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.130496 25025 rpcz_store.cc:276] 0426 13:20:23.057004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.063389 (+  6385us) service_pool.cc:225] Handling call
0426 13:20:24.118547 (+1055158us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.118767 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136699136}
W20240426 13:20:24.140679 25589 connection.cc:232] RPC call timeout handler was delayed by -2.8517126760000338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20240426 13:20:24.014191 28254 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.109637 25029 rpcz_store.cc:276] 0426 13:20:19.535813 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.090275 (+2554462us) service_pool.cc:225] Handling call
0426 13:20:22.133793 (+ 43518us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.134004 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:22.633852 (+499848us) log.cc:842] Serialized 14 byte log entry
0426 13:20:24.056521 (+1422669us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.056650 (+   129us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:24.056780 (+   130us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.056910 (+   130us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.057104 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":939839,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":124,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":166,"thread_start_us":1874566,"threads_started":2,"wal-append.queue_time_us":1440594}
I20240426 13:20:24.030359 28525 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.015197 28402 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:24.147786 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1803) took 2810 ms (client timeout 1750 ms). Trace:
W20240426 13:20:24.210055 24952 rpcz_store.cc:269] 0426 13:20:21.332242 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:23.081537 (+1749295us) service_pool.cc:225] Handling call
0426 13:20:24.138851 (+1057314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.149751 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1947) took 1215 ms. Trace:
I20240426 13:20:24.210705 25102 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:24.210991 25103 rpcz_store.cc:276] 0426 13:20:22.934372 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.131459 (+1197087us) service_pool.cc:225] Handling call
0426 13:20:24.148986 (+ 17527us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.149371 (+   385us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.149395 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.149452 (+    57us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:24.149487 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.149535 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.149633 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.149937 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.213366 28402 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.214501 24949 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:24.214568 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.215225 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 630) took 1065 ms. Trace:
I20240426 13:20:24.150559 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.216331 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 623) took 5690 ms. Trace:
I20240426 13:20:24.216562 25028 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:24.217171 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 480) took 1126 ms. Trace:
I20240426 13:20:24.216778 25100 rpcz_store.cc:276] 0426 13:20:18.525348 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.461073 (+1935725us) service_pool.cc:225] Handling call
0426 13:20:20.465437 (+  4364us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.139105 (+1673668us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.215759 (+2076654us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54003ee9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.216116 (+   357us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637242112}
I20240426 13:20:24.217643 25028 rpcz_store.cc:276] 0426 13:20:23.091170 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.136122 (+1044952us) service_pool.cc:225] Handling call
0426 13:20:24.217113 (+ 80991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.218163 25588 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.909s (SENT) [suppressed 7 similar messages]
I20240426 13:20:24.218683 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.219501 25028 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:24.150935 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.215831 24947 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:24.220626 24948 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:24.220875 28190 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:24.221259 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.152458 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 629) took 1174 ms. Trace:
I20240426 13:20:24.221972 25097 rpcz_store.cc:276] 0426 13:20:22.978154 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.146437 (+1168283us) service_pool.cc:225] Handling call
0426 13:20:24.151577 (+  5140us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.151969 (+   392us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.152000 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.152098 (+    98us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:24.152174 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.152237 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.152376 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.153661 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1189) took 1092 ms. Trace:
I20240426 13:20:24.223078 25101 rpcz_store.cc:276] 0426 13:20:23.061161 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.150135 (+1088974us) service_pool.cc:225] Handling call
0426 13:20:24.153561 (+  3426us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.154101 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.04876e+06us
I20240426 13:20:24.225946 28419 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:24.226492 28419 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:24.227345 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.155007 27916 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.228521 27916 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:24.229517 28254 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.144523 28481 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:24.157881 28485 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:24.158679 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:24.159395 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47478.6us
I20240426 13:20:24.233577 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2283) took 1250 ms. Trace:
I20240426 13:20:24.203015 27911 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:24.234002 25024 rpcz_store.cc:276] 0426 13:20:22.983411 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.138028 (+1154617us) service_pool.cc:225] Handling call
0426 13:20:24.233515 (+ 95487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.234526 28433 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:24.234704 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.236354 28548 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:24.236681 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.237769 25100 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:24.237617 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:24.159636 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.160111 28596 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.238768 28437 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:24.238901 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.240370 25103 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:24.240669 28596 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.241583 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.241238 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.247762 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:24.247974 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.248381 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.249471 28279 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:24.249504 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.149894 28440 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.161478 24950 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:24.250975 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2184) took 2096 ms (client timeout 2423 ms). Trace:
W20240426 13:20:24.251376 24950 rpcz_store.cc:269] 0426 13:20:22.155330 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.103656 (+1948326us) service_pool.cc:225] Handling call
0426 13:20:24.250886 (+147230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.162348 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:24.252655 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2284) took 1269 ms. Trace:
I20240426 13:20:24.253036 25026 rpcz_store.cc:276] 0426 13:20:22.983641 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.138391 (+1154750us) service_pool.cc:225] Handling call
0426 13:20:24.252570 (+114179us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.253646 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1937) took 3476 ms (client timeout 1626 ms). Trace:
I20240426 13:20:24.160992 24951 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:24.170066 28423 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:24.254498 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1808) took 1327 ms. Trace:
I20240426 13:20:24.254717 28423 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.170688 28592 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.254902 24951 rpcz_store.cc:276] 0426 13:20:22.926623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.120559 (+1193936us) service_pool.cc:225] Handling call
0426 13:20:24.254440 (+133881us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.173163 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1942) took 3393 ms (client timeout 1721 ms). Trace:
W20240426 13:20:24.254084 25102 rpcz_store.cc:269] 0426 13:20:20.777698 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:20.780196 (+  2498us) service_pool.cc:225] Handling call
0426 13:20:22.297390 (+1517194us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:24.253408 (+1956018us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.253545 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4406398720}
I20240426 13:20:24.144098 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 479) took 1053 ms. Trace:
I20240426 13:20:24.258297 24950 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:24.258271 25022 rpcz_store.cc:276] 0426 13:20:23.090958 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.056211 (+965253us) service_pool.cc:225] Handling call
0426 13:20:24.143984 (+ 87773us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.259191 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2160) took 6112 ms. Trace:
I20240426 13:20:24.155022 25027 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:24.177719 28325 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:24.260739 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 481) took 1156 ms. Trace:
I20240426 13:20:24.261140 25027 rpcz_store.cc:276] 0426 13:20:23.104837 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.059291 (+954454us) service_pool.cc:225] Handling call
0426 13:20:24.260663 (+201372us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.261889 28602 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.315s	user 0.003s	sys 0.000s
I20240426 13:20:24.263607 28592 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:24.179651 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 486) took 2960 ms (client timeout 1857 ms). Trace:
I20240426 13:20:24.174906 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:24.184324 28492 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:24.239074 28429 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:24.264925 28429 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.264886 28492 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.180660 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 693) took 1886 ms. Trace:
I20240426 13:20:24.266323 25623 rpcz_store.cc:276] 0426 13:20:22.293928 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.965435 (+671507us) service_pool.cc:225] Handling call
0426 13:20:24.028437 (+1063002us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.050083 (+ 21646us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.180401 (+130318us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540065e850. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.180571 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2435170688}
I20240426 13:20:24.189071 28401 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:24.188086 28460 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:24.189858 28522 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:24.215639 24949 rpcz_store.cc:276] 0426 13:20:23.748489 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.116834 (+368345us) service_pool.cc:225] Handling call
0426 13:20:24.215163 (+ 98329us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.269414 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94768.6us
I20240426 13:20:24.120172 28585 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:24.176967 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61126.1us
I20240426 13:20:24.194186 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 692) took 1900 ms. Trace:
I20240426 13:20:24.196673 28496 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:24.202222 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:24.154122 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1950) took 1214 ms. Trace:
I20240426 13:20:24.204087 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2156) took 6621 ms. Trace:
I20240426 13:20:24.204586 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2139) took 7677 ms. Trace:
I20240426 13:20:24.204893 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1944) took 1272 ms. Trace:
I20240426 13:20:24.281320 25105 rpcz_store.cc:276] 0426 13:20:22.939923 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.133265 (+1193342us) service_pool.cc:225] Handling call
0426 13:20:24.153045 (+ 19780us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.153636 (+   591us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.153680 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.153757 (+    77us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:24.153819 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.153899 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.154051 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1082,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164}
I20240426 13:20:24.143162 28498 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:24.165984 28407 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.209244 28525 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.149366 25620 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:24.221377 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2183) took 2069 ms (client timeout 2474 ms). Trace:
W20240426 13:20:24.290132 24948 rpcz_store.cc:269] 0426 13:20:22.152084 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.108515 (+1956431us) service_pool.cc:225] Handling call
0426 13:20:24.221305 (+112790us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1805056}
I20240426 13:20:24.129460 28461 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:24.236766 28548 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:24.160271 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 478) took 1069 ms. Trace:
I20240426 13:20:24.293509 25030 rpcz_store.cc:276] 0426 13:20:23.090703 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.050978 (+960275us) service_pool.cc:225] Handling call
0426 13:20:24.160208 (+109230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.238021 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:24.250135 28279 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.255174 28423 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:24.255657 25096 rpcz_store.cc:269] 0426 13:20:20.779537 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.139492 (+1359955us) service_pool.cc:225] Handling call
0426 13:20:24.172831 (+2033339us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.172963 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41453312}
I20240426 13:20:24.259301 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 626) took 6963 ms. Trace:
I20240426 13:20:24.297260 24954 rpcz_store.cc:276] 0426 13:20:17.295378 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.448233 (+1152855us) service_pool.cc:225] Handling call
0426 13:20:18.471591 (+ 23358us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:23.051593 (+4580002us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.258938 (+1207345us) spinlock_profiling.cc:243] Waited 4.5 s on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.259112 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10360612224}
I20240426 13:20:24.238803 25025 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:24.298671 25027 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:24.299468 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2270) took 4750 ms. Trace:
I20240426 13:20:24.267239 28429 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.308502 25031 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:24.257581 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.302081 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:24.271732 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:24.146879 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1953) took 1051 ms. Trace:
I20240426 13:20:24.309545 25098 rpcz_store.cc:276] 0426 13:20:23.091094 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.117763 (+1026669us) service_pool.cc:225] Handling call
0426 13:20:24.142089 (+ 24326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.172564 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.259719 24950 rpcz_store.cc:276] 0426 13:20:18.146533 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.257737 (+6111204us) service_pool.cc:225] Handling call
0426 13:20:24.258089 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.258906 (+   817us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.259088 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.310566 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2286) took 1319 ms. Trace:
I20240426 13:20:24.310937 25023 rpcz_store.cc:276] 0426 13:20:22.991008 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.137609 (+1146601us) service_pool.cc:225] Handling call
0426 13:20:24.310505 (+172896us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.189785 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.275039 25619 rpcz_store.cc:276] 0426 13:20:22.293609 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.303024 (+  9415us) service_pool.cc:225] Handling call
0426 13:20:24.193743 (+1890719us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.193974 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2742306560}
I20240426 13:20:24.258562 24952 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:24.313092 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2161) took 6166 ms. Trace:
I20240426 13:20:24.313587 24952 rpcz_store.cc:276] 0426 13:20:18.146801 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.258066 (+6111265us) service_pool.cc:225] Handling call
0426 13:20:24.258382 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.312723 (+ 54341us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.312981 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.315508 24952 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:20:24.316442 25101 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:24.321543 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 629) took 4686 ms. Trace:
I20240426 13:20:24.281574 24953 rpcz_store.cc:276] 0426 13:20:16.526783 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:16.759459 (+232676us) service_pool.cc:225] Handling call
0426 13:20:17.899995 (+1140536us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:20.341352 (+2441357us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:20.960282 (+618930us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.204179 (+3243897us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400366ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.204401 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7051363328}
I20240426 13:20:24.209391 28531 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:24.264223 25616 rpcz_store.cc:269] 0426 13:20:21.219709 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:21.219867 (+   158us) service_pool.cc:225] Handling call
0426 13:20:24.179397 (+2959530us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.179544 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4697776384}
I20240426 13:20:24.299796 25027 rpcz_store.cc:276] 0426 13:20:19.549391 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.293541 (+4744150us) service_pool.cc:225] Handling call
0426 13:20:24.298340 (+  4799us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.299202 (+   862us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.299392 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.327193 24952 rpcz_store.cc:276] 0426 13:20:19.631355 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.314570 (+4683215us) service_pool.cc:225] Handling call
0426 13:20:24.315047 (+   477us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.315946 (+   899us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.317936 (+  1990us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.282127 25099 rpcz_store.cc:276] 0426 13:20:22.932824 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.932975 (+   151us) service_pool.cc:225] Handling call
0426 13:20:22.933346 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:22.980138 (+ 46792us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.204411 (+1224273us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.204698 (+   287us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69143424}
W20240426 13:20:24.318558 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1806) took 2029 ms (client timeout 2159 ms). Trace:
I20240426 13:20:24.289772 28481 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:24.294335 28548 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:24.302589 28279 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:24.328459 24955 rpcz_store.cc:269] 0426 13:20:22.279788 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.041634 (+1761846us) service_pool.cc:225] Handling call
0426 13:20:24.309389 (+267755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.208452 28360 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:24.328400 28440 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:24.334439 25030 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:24.281397 24956 rpcz_store.cc:276] 0426 13:20:17.431913 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:18.664758 (+1232845us) service_pool.cc:225] Handling call
0426 13:20:18.898587 (+233829us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:18.898782 (+   195us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:19.076749 (+177967us) log.cc:842] Serialized 14 byte log entry
0426 13:20:21.349260 (+2272511us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b18003e7580. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb
0426 13:20:23.769581 (+2420321us) log.cc:842] Serialized 14 byte log entry
0426 13:20:23.970617 (+201036us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:23.974721 (+  4104us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:23.975075 (+   354us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.007921 (+ 32846us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.053200 (+ 45279us) inbound_call.cc:173] Queueing success response
0426 13:20:24.245791 (+192591us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b18003e7580. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":50346,"prepare.queue_time_us":1136280,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":9591,"raft.queue_time_us":256631,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":50427,"spinlock_wait_cycles":20457216,"thread_start_us":3240539,"threads_started":3,"wal-append.queue_time_us":834306,"wal-append.run_cpu_time_us":1057,"wal-append.run_wall_time_us":1005802}
I20240426 13:20:23.072340 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:24.308246 28492 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:24.340553 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 122436us
I20240426 13:20:24.346973 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:24.345373 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 155845us
I20240426 13:20:24.349973 25025 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240426 13:20:24.352988 25022 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:24.355124 25031 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:24.355772 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2275) took 3665 ms. Trace:
I20240426 13:20:24.355947 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2282) took 1372 ms. Trace:
I20240426 13:20:24.356509 25031 rpcz_store.cc:276] 0426 13:20:22.983122 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.321357 (+1338235us) service_pool.cc:225] Handling call
0426 13:20:24.354854 (+ 33497us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.355647 (+   793us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.355874 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.356948 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.336190 28344 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:24.358350 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2267) took 4810 ms. Trace:
I20240426 13:20:24.353718 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2276) took 3657 ms. Trace:
W20240426 13:20:24.360601 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 51217.9us
I20240426 13:20:24.360957 25030 rpcz_store.cc:276] 0426 13:20:20.691008 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.316262 (+3625254us) service_pool.cc:225] Handling call
0426 13:20:24.334198 (+ 17936us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.348645 (+ 14447us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.348844 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.360654 25028 rpcz_store.cc:276] 0426 13:20:19.541276 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.289369 (+4748093us) service_pool.cc:225] Handling call
0426 13:20:24.324915 (+ 35546us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.344874 (+ 19959us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.344921 (+    47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.344979 (+    58us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:24.351821 (+  6842us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.351902 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.352021 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.356341 25024 rpcz_store.cc:276] 0426 13:20:20.690713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.287426 (+3596713us) service_pool.cc:225] Handling call
0426 13:20:24.342807 (+ 55381us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.355306 (+ 12499us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.355344 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.355410 (+    66us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:24.355464 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.355539 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.355686 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.354188 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2277) took 3659 ms. Trace:
I20240426 13:20:24.363855 25025 rpcz_store.cc:276] 0426 13:20:20.691270 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.298942 (+3607672us) service_pool.cc:225] Handling call
0426 13:20:24.329745 (+ 30803us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.350436 (+ 20691us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.350663 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.372226 28429 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:24.363526 28401 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:24.375181 28401 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.375573 28401 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:24.343906 28440 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:24.375756 28407 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.376358 28344 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.386180 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:24.387589 24953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:20:24.376971 28440 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:24.390952 24995 leader_election.cc:334] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Remote error: Timed out: Call waited in the queue past client deadline
I20240426 13:20:24.391508 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:24.392329 28360 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:24.392900 28360 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:24.393913 28360 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.395195 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2274) took 4002 ms. Trace:
I20240426 13:20:24.395557 25029 rpcz_store.cc:276] 0426 13:20:20.392767 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.294847 (+3902080us) service_pool.cc:225] Handling call
0426 13:20:24.336869 (+ 42022us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.394874 (+ 58005us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.394903 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.394940 (+    37us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:24.394975 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.395026 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.395133 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.396839 25028 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:24.397316 25104 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:24.400148 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:24.400266 28408 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:24.400874 28461 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:24.401456 28461 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.400993 28408 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.402506 24955 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:24.403224 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2171) took 4593 ms. Trace:
I20240426 13:20:24.403584 24955 rpcz_store.cc:276] 0426 13:20:19.809597 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.342807 (+4533210us) service_pool.cc:225] Handling call
0426 13:20:24.402286 (+ 59479us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.402920 (+   634us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.403160 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.398604 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 662) took 2334 ms. Trace:
I20240426 13:20:24.404474 25620 rpcz_store.cc:276] 0426 13:20:22.064686 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:22.308617 (+243931us) service_pool.cc:225] Handling call
0426 13:20:22.308883 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.316936 (+2008053us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.398334 (+ 81398us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b5400380c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.398546 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4231125376}
I20240426 13:20:24.380489 28540 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:24.381371 28394 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20240426 13:20:24.406078 28540 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:24.406770 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59798.4us
W20240426 13:20:24.381119 28595 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.190s	user 0.000s	sys 0.002s
I20240426 13:20:24.407930 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2273) took 4015 ms. Trace:
I20240426 13:20:24.408289 28419 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.408357 25026 rpcz_store.cc:276] 0426 13:20:20.392454 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.303215 (+3910761us) service_pool.cc:225] Handling call
0426 13:20:24.353297 (+ 50082us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.407569 (+ 54272us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.407604 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.407666 (+    62us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:24.407737 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.407789 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.407883 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.397817 28062 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:24.409759 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2175) took 4553 ms. Trace:
I20240426 13:20:24.380407 28583 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20240426 13:20:24.388490 24953 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:20:24.412348 24953 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:24.413134 28366 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:24.413636 28366 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.377390 28531 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.397796 28444 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:24.417248 28455 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.308s	user 0.005s	sys 0.009s
I20240426 13:20:24.417845 28444 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:24.401887 28461 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:24.409716 28062 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.424176 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:24.424772 28062 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.425963 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:24.410501 28419 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:24.417676 28455 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.431622 28580 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:24.432103 28580 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.432435 28580 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:24.432751 28455 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:24.385524 24956 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:20:24.433970 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2172) took 4616 ms. Trace:
I20240426 13:20:24.434386 24956 rpcz_store.cc:276] 0426 13:20:19.818061 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.353614 (+4535553us) service_pool.cc:225] Handling call
0426 13:20:24.385153 (+ 31539us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.433644 (+ 48491us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.433883 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.435787 24956 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:24.436705 28412 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:24.436470 28055 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:24.437083 28412 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.437144 25031 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:24.379565 28437 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:24.438118 28412 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.438910 25100 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:24.387629 28344 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:24.430169 28577 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:24.407300 28540 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.477682 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:24.468340 25030 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:24.479470 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:24.453229 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1604) took 4908 ms (client timeout 1994 ms). Trace:
I20240426 13:20:24.485802 25098 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:24.482039 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:24.465996 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.488097 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.463816 28394 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:24.489071 28394 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.465512 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:24.488824 25096 raft_consensus.cc:2354] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:20:24.437983 28055 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:24.490437 28055 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:24.490692 28577 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:24.490883 28055 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:24.467428 25100 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:24.464588 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 99862.6us
I20240426 13:20:24.491765 28577 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.493171 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.463374 28395 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:24.493561 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65196.4us
I20240426 13:20:24.494715 28395 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:24.495220 28395 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:24.495659 28395 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:24.439898 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2178) took 3935 ms. Trace:
I20240426 13:20:24.496511 24956 rpcz_store.cc:276] 0426 13:20:20.504600 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.435303 (+3930703us) service_pool.cc:225] Handling call
0426 13:20:24.435632 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.439625 (+  3993us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.439835 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.444530 24951 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:24.498248 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.499087 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2159) took 6352 ms. Trace:
I20240426 13:20:24.499508 24951 rpcz_store.cc:276] 0426 13:20:18.146259 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.257382 (+6111123us) service_pool.cc:225] Handling call
0426 13:20:24.257866 (+   484us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.498316 (+240450us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.498860 (+   544us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:24.499016 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":428418816}
I20240426 13:20:24.464617 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:24.463271 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63236.5us
I20240426 13:20:24.490063 25099 raft_consensus.cc:2354] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:24.439767 28412 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:24.506834 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:24.501013 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:24.495349 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:24.410370 24955 rpcz_store.cc:276] 0426 13:20:19.855858 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.404468 (+4548610us) service_pool.cc:225] Handling call
0426 13:20:24.404844 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.405055 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.405078 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:24.405138 (+    60us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:24.409543 (+  4405us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.409602 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:24.409693 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.426323 28062 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:24.751221 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.752239 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:24.438139 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 476) took 1349 ms. Trace:
I20240426 13:20:24.568980 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:24.618950 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:25.344972 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 279) took 19912 ms (client timeout 15000 ms). Trace:
I20240426 13:20:24.609256 25102 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
W20240426 13:20:25.456619 25022 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.077s	user 0.031s	sys 0.032s
I20240426 13:20:24.752494 25025 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
W20240426 13:20:24.565675 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64605.8us
W20240426 13:20:24.486073 25104 rpcz_store.cc:269] 0426 13:20:19.544516 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:19.692575 (+148059us) service_pool.cc:225] Handling call
0426 13:20:24.452985 (+4760410us) spinlock_profiling.cc:243] Waited 4.62 s on lock 0x7b54003b0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:24.453142 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10634223360}
I20240426 13:20:24.672204 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:24.548900 24953 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.460711 25031 rpcz_store.cc:276] 0426 13:20:23.089140 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.365259 (+1276119us) service_pool.cc:225] Handling call
0426 13:20:24.436923 (+ 71664us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.437848 (+   925us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.438075 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:24.501298 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74470.8us
W20240426 13:20:25.466594 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 971228us
I20240426 13:20:25.463474 25022 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:25.475229 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2289) took 1398 ms. Trace:
W20240426 13:20:25.194751 28279 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.843s	user 0.011s	sys 0.000s
W20240426 13:20:24.949407 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 558456us
I20240426 13:20:25.479197 24953 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:24.968638 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:25.475783 25022 rpcz_store.cc:276] 0426 13:20:24.077223 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.284599 (+207376us) service_pool.cc:225] Handling call
0426 13:20:25.475149 (+1190550us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:25.357630 28580 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.925s	user 0.010s	sys 0.000s
W20240426 13:20:25.342737 28055 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.851s	user 0.011s	sys 0.000s
I20240426 13:20:24.875442 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.874897 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.477866 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1962) took 1223 ms. Trace:
I20240426 13:20:24.617333 28408 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:24.935485 28481 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.006s	sys 0.019s
I20240426 13:20:24.493736 28577 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:25.502933 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.502969 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:25.530565 25024 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:25.480949 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1635) took 1036 ms. Trace:
I20240426 13:20:24.462116 28366 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:25.508731 25102 rpcz_store.cc:276] 0426 13:20:24.254319 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.256948 (+  2629us) service_pool.cc:225] Handling call
0426 13:20:25.477691 (+1220743us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:25.477800 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":689882880}
I20240426 13:20:25.478358 24952 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 12. Ops: []
I20240426 13:20:25.508242 25028 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:25.511065 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2169) took 5721 ms. Trace:
I20240426 13:20:24.490089 28394 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.572762 28492 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.939s
I20240426 13:20:25.520062 25624 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:25.520217 28412 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:25.188665 24851 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 7, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 7 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.521601 28429 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:25.521908 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:25.522444 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.521607 24951 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:20:25.550627 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1788) took 4861 ms. Trace:
I20240426 13:20:25.551000 24951 rpcz_store.cc:276] 0426 13:20:20.691678 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.520937 (+4829259us) service_pool.cc:225] Handling call
0426 13:20:25.521348 (+   411us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:25.550326 (+ 28978us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.550573 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.102058 28498 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:25.556679 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:25.523519 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:25.548358 25025 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:25.529170 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:24.835551 28496 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.546s	user 0.008s	sys 0.000s
I20240426 13:20:25.531172 25030 rpcz_store.cc:276] 0426 13:20:24.431403 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.750835 (+319432us) service_pool.cc:225] Handling call
0426 13:20:25.467666 (+716831us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.535079 24953 rpcz_store.cc:276] 0426 13:20:19.769463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.339515 (+4570052us) service_pool.cc:225] Handling call
0426 13:20:25.479703 (+1140188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.529445 25029 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:25.540642 25026 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:20:25.563902 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59302.9us
W20240426 13:20:25.530071 28481 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.014s	sys 0.031s
I20240426 13:20:24.572008 28423 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.548915 25099 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:25.601749 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.602103 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1614) took 1112 ms. Trace:
I20240426 13:20:25.525013 28580 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.572767 28401 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.752512 28461 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:24.743523 28548 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:25.563443 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:24.745795 28395 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:24.451692 28440 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.168385 28049 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.785s
I20240426 13:20:25.585521 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 632) took 1812 ms. Trace:
I20240426 13:20:25.669931 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.584645 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:25.589871 25615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.001s	sys 0.017s
W20240426 13:20:25.671474 25615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.216s	user 0.001s	sys 0.022s
I20240426 13:20:25.590314 24955 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:25.590819 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:25.552475 28412 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:25.592813 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.670652 24951 rpcz_store.cc:276] 0426 13:20:23.750007 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.551810 (+1801803us) service_pool.cc:225] Handling call
0426 13:20:25.556734 (+  4924us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.562480 (+  5746us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.562510 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.562560 (+    50us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:25.562597 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.562655 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.562765 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.671224 25620 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:25.675422 28580 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.675556 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 634) took 1924 ms. Trace:
I20240426 13:20:25.677524 28401 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:25.679275 28395 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:25.676101 24951 rpcz_store.cc:276] 0426 13:20:23.751007 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.673334 (+1922327us) service_pool.cc:225] Handling call
0426 13:20:25.674853 (+  1519us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.675188 (+   335us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.675218 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.675265 (+    47us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:25.675313 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.675372 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.675500 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.673568 28412 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:25.684734 25097 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:25.685398 28423 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.686040 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.686784 25098 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
W20240426 13:20:25.687314 28616 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20240426 13:20:25.688491 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:25.689023 28580 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:25.689525 28580 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:25.690018 28580 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:25.502753 28279 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.690642 28560 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:25.691184 28560 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:24.816339 28585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.010s	sys 0.021s
I20240426 13:20:25.595343 25022 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:25.587908 24851 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.523322 28055 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:25.601256 25096 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:25.602541 25099 rpcz_store.cc:276] 0426 13:20:24.530291 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.561647 (+ 31356us) service_pool.cc:225] Handling call
0426 13:20:25.602031 (+1040384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:24.572515 28592 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.965s
I20240426 13:20:25.602653 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.699038 25103 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240426 13:20:25.598158 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.586648 28254 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:25.699716 25104 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:25.700321 24948 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:25.477414 28360 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:25.669200 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:25.669898 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58474.4us
I20240426 13:20:25.702134 25617 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:25.702464 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:25.670641 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:25.703238 24956 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:25.057754 27810 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:25.692956 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2182) took 3548 ms. Trace:
I20240426 13:20:25.692644 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2280) took 3506 ms. Trace:
I20240426 13:20:25.692862 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2285) took 2702 ms. Trace:
I20240426 13:20:25.708495 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.708163 24955 rpcz_store.cc:276] 0426 13:20:22.144983 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.541756 (+3396773us) service_pool.cc:225] Handling call
0426 13:20:25.590083 (+ 48327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.692703 (+102620us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.692898 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:25.698554 28585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.424s	user 0.013s	sys 0.027s
I20240426 13:20:25.712286 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:25.462572 24853 rpcz_store.cc:269] 0426 13:20:05.432539 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.452160 (+ 19621us) service_pool.cc:225] Handling call
0426 13:20:25.344641 (+19892481us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:16.508117 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:16.513015 (+  4898us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:16.513073 (+    58us) write_op.cc:427] Acquired schema lock
0426 13:20:16.513089 (+    16us) tablet.cc:660] Decoding operations
0426 13:20:16.588911 (+ 75822us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:16.589008 (+    97us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:16.589036 (+    28us) tablet.cc:683] Acquiring locks for 20 operations
0426 13:20:16.590555 (+  1519us) tablet.cc:699] Row locks acquired
0426 13:20:16.590591 (+    36us) write_op.cc:260] PREPARE: finished
0426 13:20:16.590778 (+   187us) write_op.cc:270] Start()
0426 13:20:16.590921 (+   143us) write_op.cc:275] Timestamp: P: 1714137616590707 usec, L: 0
0426 13:20:16.590951 (+    30us) op_driver.cc:331] REPLICATION: starting
0426 13:20:16.591671 (+   720us) log.cc:842] Serialized 7750 byte log entry
0426 13:20:16.814355 (+222684us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8d3cee2e 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b
0426 13:20:18.221470 (+1407115us) op_driver.cc:444] REPLICATION: finished
0426 13:20:19.239300 (+1017830us) write_op.cc:300] APPLY: starting
0426 13:20:19.268189 (+ 28889us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:19.660461 (+392272us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:19.660508 (+    47us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:19.815327 (+154819us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:20.467164 (+651837us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0426 13:20:20.498271 (+ 31107us) write_op.cc:311] APPLY: finished
0426 13:20:20.742014 (+243743us) log.cc:842] Serialized 139 byte log entry
0426 13:20:22.463520 (+1721506us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:22.579341 (+115821us) write_op.cc:446] Released schema lock
0426 13:20:22.594406 (+ 15065us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:22.961987 (+367581us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:22.962018 (+    31us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2128288,"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":862249,"apply.run_cpu_time_us":90314,"apply.run_wall_time_us":3794758,"num_ops":20,"prepare.queue_time_us":303233,"prepare.run_cpu_time_us":50632,"prepare.run_wall_time_us":757775,"raft.queue_time_us":537310,"raft.run_cpu_time_us":55472,"raft.run_wall_time_us":632452,"replication_time_us":1603480,"spinlock_wait_cycles":18184576,"thread_start_us":2815953,"threads_started":5,"wal-append.queue_time_us":2011921,"wal-append.run_cpu_time_us":24933,"wal-append.run_wall_time_us":2717970}]]}
W20240426 13:20:25.694938 25069 connection.cc:232] RPC call timeout handler was delayed by -2.2891266970000288s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20240426 13:20:25.689961 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:25.704989 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.010s	sys 0.041s
W20240426 13:20:25.716737 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.010s	sys 0.041s
W20240426 13:20:25.689024 28618 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:20:25.737196 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:25.720706 28055 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.713580 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:25.692008 28279 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.667574 28498 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:25.721120 25101 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:25.721859 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.708901 25027 rpcz_store.cc:276] 0426 13:20:22.990699 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.327929 (+1337230us) service_pool.cc:225] Handling call
0426 13:20:24.334658 (+  6729us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:24.335164 (+   506us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.335197 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:24.335767 (+   570us) log.cc:842] Serialized 26 byte log entry
0426 13:20:24.827078 (+491311us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:24.827135 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.827462 (+   327us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:25.692697 (+865235us) raft_consensus.cc:1671] finished
0426 13:20:25.692716 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.692818 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1462,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":7581,"thread_start_us":490752,"threads_started":1,"wal-append.queue_time_us":1337689}
I20240426 13:20:25.752974 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2165) took 7328 ms. Trace:
I20240426 13:20:25.758060 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.757679 28498 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:25.758639 28440 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.759351 28440 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:25.759807 28585 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:25.758749 25027 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:25.729079 28548 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:25.729069 28412 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.720360 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:25.744136 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 668) took 1238 ms. Trace:
I20240426 13:20:25.748613 25617 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:25.764933 25619 rpcz_store.cc:276] 0426 13:20:24.506050 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.560931 (+ 54881us) service_pool.cc:225] Handling call
0426 13:20:25.743908 (+1182977us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:25.744074 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":479982976}
I20240426 13:20:25.749975 24955 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:25.766095 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 637) took 1712 ms. Trace:
I20240426 13:20:25.766613 24955 rpcz_store.cc:276] 0426 13:20:24.053707 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.718514 (+1664807us) service_pool.cc:225] Handling call
0426 13:20:25.745148 (+ 26634us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.765773 (+ 20625us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.766030 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.752568 25031 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:20:25.752957 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.753407 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.753715 28279 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:25.754031 25030 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:25.754371 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.768582 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:25.768838 25029 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:25.769655 25022 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:25.769485 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:25.754918 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.770648 25025 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:20:25.755225 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.753453 28049 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:25.758194 28408 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:25.708657 25023 rpcz_store.cc:276] 0426 13:20:22.186518 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.311589 (+2125071us) service_pool.cc:225] Handling call
0426 13:20:24.312069 (+   480us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:24.312555 (+   486us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.312582 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:24.313017 (+   435us) log.cc:842] Serialized 26 byte log entry
0426 13:20:24.352244 (+ 39227us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:24.352282 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:24.352351 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:25.692444 (+1340093us) raft_consensus.cc:1671] finished
0426 13:20:25.692463 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.692593 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1852,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":113,"thread_start_us":38850,"threads_started":1,"wal-append.queue_time_us":1195822}
I20240426 13:20:25.703596 28254 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:25.758078 24948 rpcz_store.cc:276] 0426 13:20:18.425005 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.302286 (+5877281us) service_pool.cc:225] Handling call
0426 13:20:24.339589 (+ 37303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.700726 (+1361137us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.752755 (+ 52029us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400102054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:25.752923 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3118606080}
I20240426 13:20:25.770154 25098 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
W20240426 13:20:25.763749 24921 connection.cc:232] RPC call timeout handler was delayed by -4.280307353000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20240426 13:20:25.774602 24921 connection.cc:232] RPC call timeout handler was delayed by -3.9509073530000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1867s for a response
W20240426 13:20:25.775139 24921 connection.cc:232] RPC call timeout handler was delayed by -3.8366073530000904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19940000000000002s for a response
W20240426 13:20:25.775516 24921 connection.cc:232] RPC call timeout handler was delayed by -3.82400735300007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20240426 13:20:25.775909 24921 connection.cc:232] RPC call timeout handler was delayed by -3.819933836000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27040000000000003s for a response
I20240426 13:20:25.760291 28585 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:25.751842 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2179) took 5228 ms. Trace:
W20240426 13:20:25.776505 24921 connection.cc:232] RPC call timeout handler was delayed by -3.7187073530000134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
W20240426 13:20:25.785838 24921 connection.cc:232] RPC call timeout handler was delayed by -3.717807353000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
W20240426 13:20:25.786350 24921 connection.cc:232] RPC call timeout handler was delayed by -3.6165338360001442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29300000000000004s for a response
W20240426 13:20:25.786784 24921 connection.cc:232] RPC call timeout handler was delayed by -3.5427338360001386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3012s for a response
I20240426 13:20:25.778582 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:25.787295 24921 connection.cc:232] RPC call timeout handler was delayed by -3.155307353000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2751s for a response
I20240426 13:20:25.772966 28049 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:25.773433 28408 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:25.779625 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.784387 25099 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10.
W20240426 13:20:25.787717 24921 connection.cc:232] RPC call timeout handler was delayed by -2.8745073530000127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3063s for a response
I20240426 13:20:25.785526 24956 rpcz_store.cc:276] 0426 13:20:20.523304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.572829 (+4049525us) service_pool.cc:225] Handling call
0426 13:20:24.573195 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.712453 (+1139258us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.751580 (+ 39127us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:25.751757 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2598396288}
I20240426 13:20:25.793170 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:25.793967 25027 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:25.784507 24851 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 9, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.801535 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:25.799711 25025 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:25.806033 25023 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:25.806891 25029 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:25.777315 28254 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:25.809092 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 639) took 1752 ms. Trace:
I20240426 13:20:25.809633 24948 rpcz_store.cc:276] 0426 13:20:24.057031 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.774109 (+1717078us) service_pool.cc:225] Handling call
0426 13:20:25.794841 (+ 20732us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.808595 (+ 13754us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.808644 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.808726 (+    82us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:25.808769 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.808850 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.809012 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.800416 28408 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.812728 24851 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.814878 24851 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.815598 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2168) took 6068 ms. Trace:
I20240426 13:20:25.759933 28440 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.817544 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 641) took 1740 ms. Trace:
I20240426 13:20:25.789395 28049 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:25.821988 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1608) took 1358 ms. Trace:
I20240426 13:20:25.822292 25097 rpcz_store.cc:276] 0426 13:20:24.464093 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.464283 (+   190us) service_pool.cc:225] Handling call
0426 13:20:25.821844 (+1357561us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540001ccd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:25.821944 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2720420096}
I20240426 13:20:25.818938 24956 rpcz_store.cc:276] 0426 13:20:24.077015 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.792749 (+1715734us) service_pool.cc:225] Handling call
0426 13:20:25.808230 (+ 15481us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.817134 (+  8904us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.817165 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.817225 (+    60us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:25.817270 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.817335 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.817468 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.818413 24952 rpcz_store.cc:276] 0426 13:20:19.746972 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.341022 (+4594050us) service_pool.cc:225] Handling call
0426 13:20:24.427674 (+ 86652us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.541602 (+1113928us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.815299 (+273697us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:25.815513 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2387109248}
I20240426 13:20:25.770725 27810 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20240426 13:20:25.831939 25030 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:25.775702 28360 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.779311 28279 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:25.845995 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 9, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.779834 28412 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:25.848071 25028 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:25.856618 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2185) took 1353 ms. Trace:
I20240426 13:20:25.892439 24956 rpcz_store.cc:276] 0426 13:20:24.503730 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.829134 (+1325404us) service_pool.cc:225] Handling call
0426 13:20:25.844199 (+ 15065us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.856118 (+ 11919us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.856179 (+    61us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.856246 (+    67us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:25.856307 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.856390 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.856547 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.850081 28461 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:25.857955 24851 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 9. New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.861611 28560 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:25.863411 28394 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:25.854486 28585 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:25.869418 28440 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:25.893494 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97323.8us
I20240426 13:20:25.894038 28461 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:25.897864 28461 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:25.899009 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:25.898852 28461 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:25.891052 28613 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.372s	user 0.009s	sys 0.004s
I20240426 13:20:25.903977 24956 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:25.906396 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2187) took 1399 ms. Trace:
I20240426 13:20:25.906785 24956 rpcz_store.cc:276] 0426 13:20:24.506620 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.893508 (+1386888us) service_pool.cc:225] Handling call
0426 13:20:25.901296 (+  7788us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.906056 (+  4760us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.906314 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:25.902984 28440 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:25.915663 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:25.907917 28279 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:25.909072 28461 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:25.925956 25624 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:25.931080 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 669) took 1424 ms. Trace:
I20240426 13:20:25.932775 28254 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:25.935178 28254 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:25.935024 28437 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:25.931528 25624 rpcz_store.cc:276] 0426 13:20:24.506389 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.560859 (+ 54470us) service_pool.cc:225] Handling call
0426 13:20:25.931004 (+1370145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34560}
I20240426 13:20:25.942027 25102 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
W20240426 13:20:25.945369 24949 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.593s	user 0.022s	sys 0.034s
I20240426 13:20:25.916256 28440 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:25.945838 24949 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.596s	user 0.022s	sys 0.035s
I20240426 13:20:25.927063 28585 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:25.904120 24851 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 8, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:25.929472 28548 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:25.948901 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.196s	user 0.005s	sys 0.056s
W20240426 13:20:25.954188 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.202s	user 0.005s	sys 0.056s
I20240426 13:20:25.943351 28360 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:25.948733 27911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.314s	user 0.022s	sys 0.034s
I20240426 13:20:25.947569 28440 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:25.971299 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2167) took 7541 ms. Trace:
I20240426 13:20:25.972869 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:20:25.973160 27911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.338s	user 0.023s	sys 0.034s
I20240426 13:20:25.974792 24950 rpcz_store.cc:276] 0426 13:20:18.429962 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.310512 (+5880550us) service_pool.cc:225] Handling call
0426 13:20:24.310947 (+   435us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.311249 (+   302us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:24.719413 (+408164us) log.cc:842] Serialized 14 byte log entry
0426 13:20:25.970963 (+1251550us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:25.971014 (+    51us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:25.971061 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:25.971123 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:25.971244 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1586554,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":8878,"thread_start_us":1609974,"threads_started":2}
I20240426 13:20:25.954252 28580 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.020781 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 8, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 8 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:26.030786 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.035s	sys 0.050s
W20240426 13:20:26.031277 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.035s	sys 0.050s
I20240426 13:20:26.040010 24851 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:26.042858 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2166) took 7613 ms. Trace:
I20240426 13:20:26.052203 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:26.111786 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 9, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:26.051472 24954 rpcz_store.cc:276] 0426 13:20:18.429134 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.298433 (+5869299us) service_pool.cc:225] Handling call
0426 13:20:24.338938 (+ 40505us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:24.439896 (+100958us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:24.814123 (+374227us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.039512 (+1225389us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:26.039568 (+    56us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:26.042573 (+  3005us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.042669 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.042798 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1458380,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":7687,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":2197,"thread_start_us":1265389,"threads_started":2}
W20240426 13:20:26.120463 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76992.4us
I20240426 13:20:26.121841 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.122889 28437 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:26.093622 28548 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:26.132776 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:26.126988 28548 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:26.155807 28437 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:26.158681 28548 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:26.165563 28580 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:26.203756 28437 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:26.215808 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91564.9us
I20240426 13:20:26.214625 28437 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:26.226843 24947 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.017s	sys 0.025s
W20240426 13:20:26.227172 24947 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.017s	sys 0.025s
I20240426 13:20:26.232725 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:26.230939 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.000s	sys 0.024s
W20240426 13:20:26.238306 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.000s	sys 0.025s
W20240426 13:20:26.238670 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.61808e+06us
W20240426 13:20:26.226608 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.015s	sys 0.025s
W20240426 13:20:26.239822 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.016s	sys 0.025s
W20240426 13:20:26.240199 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 722116us
I20240426 13:20:26.233940 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2163) took 7922 ms. Trace:
I20240426 13:20:26.263536 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:26.264477 25617 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:26.267190 25624 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:26.241386 24947 rpcz_store.cc:276] 0426 13:20:18.311295 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.285244 (+5973949us) service_pool.cc:225] Handling call
0426 13:20:24.340074 (+ 54830us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:24.415192 (+ 75118us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.415226 (+    34us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:24.454119 (+ 38893us) log.cc:842] Serialized 600 byte log entry
0426 13:20:24.789436 (+335317us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:26.229727 (+1440291us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.233582 (+  3855us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.233659 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.233700 (+    41us) raft_consensus.cc:1671] finished
0426 13:20:26.233711 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.233877 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1108968,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":8938,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":1087830,"threads_started":2,"wal-append.queue_time_us":1495976}
I20240426 13:20:26.298540 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:20:26.296104 25622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.007s	sys 0.033s
W20240426 13:20:26.302987 25622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.007s	sys 0.033s
W20240426 13:20:26.318431 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.000s	sys 0.029s
W20240426 13:20:26.320354 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.000s	sys 0.030s
I20240426 13:20:26.275059 28279 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:26.329387 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:26.331704 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 114917us
I20240426 13:20:26.334074 24954 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:26.279870 28461 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:26.336138 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:26.341315 24995 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.011s	sys 0.027s
W20240426 13:20:26.346422 24995 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.011s	sys 0.027s
W20240426 13:20:26.346804 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 823986us
W20240426 13:20:26.347313 25616 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.002s	sys 0.035s
W20240426 13:20:26.347699 25616 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.002s	sys 0.035s
I20240426 13:20:26.283311 28560 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
W20240426 13:20:26.344072 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.016s	sys 0.021s
W20240426 13:20:26.357065 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.016s	sys 0.021s
W20240426 13:20:26.358858 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 656451us
I20240426 13:20:26.318018 28440 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.360731 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.361476 25102 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:26.363767 24953 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.008s	sys 0.031s
W20240426 13:20:26.364202 25100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.000s	sys 0.039s
W20240426 13:20:26.362694 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.007s	sys 0.025s
W20240426 13:20:26.368827 25100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.000s	sys 0.039s
W20240426 13:20:26.367162 25620 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.024s	sys 0.035s
W20240426 13:20:26.369027 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.007s	sys 0.028s
W20240426 13:20:26.368299 25615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.009s	sys 0.030s
W20240426 13:20:26.370126 25615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.009s	sys 0.030s
W20240426 13:20:26.362649 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.017s	sys 0.041s
W20240426 13:20:26.370800 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.017s	sys 0.042s
W20240426 13:20:26.371161 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 849274us
W20240426 13:20:26.368636 24953 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.009s	sys 0.031s
W20240426 13:20:26.372107 24951 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.019s	sys 0.037s
W20240426 13:20:26.373193 24951 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.019s	sys 0.037s
W20240426 13:20:26.369590 25620 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.024s	sys 0.035s
I20240426 13:20:26.376067 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:26.365970 28540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.027s	sys 0.030s
W20240426 13:20:26.378866 28540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.028s	sys 0.031s
I20240426 13:20:26.372248 24921 leader_election.cc:398] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:20:26.382220 24921 leader_election.cc:401] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:26.376819 25096 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:26.351703 28461 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:26.389227 25069 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.005s	sys 0.032s
I20240426 13:20:26.345431 28062 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
W20240426 13:20:26.386595 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.023s	sys 0.020s
I20240426 13:20:26.391778 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.394120 25098 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:26.394868 24952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.014s	sys 0.026s
W20240426 13:20:26.395411 25069 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.005s	sys 0.032s
W20240426 13:20:26.392200 24948 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.009s	sys 0.037s
W20240426 13:20:26.395001 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.024s	sys 0.020s
W20240426 13:20:26.395453 24952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.014s	sys 0.026s
W20240426 13:20:26.396574 24948 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.009s	sys 0.037s
W20240426 13:20:26.395910 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 618404us
I20240426 13:20:26.389272 28279 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:26.403841 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.401262 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.404078 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.408298 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.410140 24954 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:26.417107 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2162) took 8113 ms. Trace:
I20240426 13:20:26.420249 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.421037 25103 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:26.386123 28437 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.423194 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.407616 28279 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:26.426434 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.411635 28254 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 13:20:26.430022 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77393us
I20240426 13:20:26.429925 28580 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:26.414780 28423 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:26.427456 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.435117 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:26.435217 25104 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:26.417609 24949 rpcz_store.cc:276] 0426 13:20:18.298846 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.268693 (+5969847us) service_pool.cc:225] Handling call
0426 13:20:24.269250 (+   557us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:24.269854 (+   604us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.269898 (+    44us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:24.270659 (+   761us) log.cc:842] Serialized 78 byte log entry
0426 13:20:24.340003 (+ 69344us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:25.946372 (+1606369us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.409678 (+463306us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.411754 (+  2076us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.411860 (+   106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.411933 (+    73us) raft_consensus.cc:1671] finished
0426 13:20:26.411945 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.412107 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11686,"prepare.queue_time_us":1555910,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":13888,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":329,"thread_start_us":2103470,"threads_started":3,"wal-append.queue_time_us":88916,"wal-append.run_cpu_time_us":2086,"wal-append.run_wall_time_us":1096896}
I20240426 13:20:26.437673 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.432801 24954 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:26.444118 28642 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:26.444731 28642 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:26.445214 28642 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:26.445564 28635 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:20:26.447101 28625 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
W20240426 13:20:26.448511 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48041us
I20240426 13:20:26.447943 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 638) took 2391 ms. Trace:
I20240426 13:20:26.450354 24955 rpcz_store.cc:276] 0426 13:20:24.056301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.768101 (+1711800us) service_pool.cc:225] Handling call
0426 13:20:25.787961 (+ 19860us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:25.824402 (+ 36441us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.824434 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:25.824882 (+   448us) log.cc:842] Serialized 26 byte log entry
0426 13:20:26.397020 (+572138us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:26.397064 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.397125 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.447727 (+ 50602us) raft_consensus.cc:1671] finished
0426 13:20:26.447747 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.447869 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5182,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":46976,"thread_start_us":548431,"threads_started":1,"wal-append.queue_time_us":607726}
I20240426 13:20:26.452073 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:26.452739 28394 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.556s	user 0.010s	sys 0.000s
I20240426 13:20:26.452677 24955 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:26.454136 28585 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.000s
I20240426 13:20:26.454991 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1190) took 2201 ms. Trace:
I20240426 13:20:26.455572 25105 rpcz_store.cc:276] 0426 13:20:24.254030 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.286678 (+ 32648us) service_pool.cc:225] Handling call
0426 13:20:26.454509 (+2167831us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:26.454910 (+   401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4887541632}
I20240426 13:20:26.457036 28062 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:26.457638 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 690) took 4980 ms. Trace:
I20240426 13:20:26.458056 25615 rpcz_store.cc:276] 0426 13:20:21.477679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:21.477877 (+   198us) service_pool.cc:225] Handling call
0426 13:20:21.589065 (+111188us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:22.016417 (+427352us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:24.247429 (+2231012us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b180045de40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789
0426 13:20:24.249417 (+  1988us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b180045de40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:20:24.359528 (+110111us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.375686 (+2016158us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:26.407784 (+ 32098us) log.cc:842] Serialized 26 byte log entry
0426 13:20:26.415917 (+  8133us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:26.415957 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.416050 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.457413 (+ 41363us) raft_consensus.cc:1671] finished
0426 13:20:26.457432 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.457564 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12045,"prepare.queue_time_us":2020886,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":105,"raft.queue_time_us":493334,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":13391,"spinlock_wait_cycles":31415424,"thread_start_us":2407191,"threads_started":4,"wal-append.queue_time_us":1499974}
I20240426 13:20:26.434355 28254 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:26.461529 28631 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:26.462498 27911 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:26.463342 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 701) took 1978 ms. Trace:
I20240426 13:20:26.486011 28631 consensus_queue.cc:1225] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:26.454912 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.488468 24955 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:26.465118 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.492714 25098 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:26.469014 28461 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:26.469617 24956 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:26.472693 28440 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:26.476594 28628 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20240426 13:20:26.477283 28437 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.438716 28408 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:26.431829 28279 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:26.518790 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 87605us
I20240426 13:20:26.478099 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.522930 24954 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:26.522966 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.524065 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.525180 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.422412 28630 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:26.525836 28630 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:26.526796 28630 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.403003 28633 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20240426 13:20:26.483774 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:26.528525 28633 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
W20240426 13:20:26.528800 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45107.8us
I20240426 13:20:26.529911 28630 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:26.531186 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.485317 28254 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:26.517443 28440 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:26.552366 28440 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:26.553193 28440 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.554499 28440 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:26.515422 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84148.6us
I20240426 13:20:26.486140 27911 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:26.516324 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:26.557826 24921 leader_election.cc:398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:26.558269 24921 leader_election.cc:401] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20240426 13:20:26.519573 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 219848us
I20240426 13:20:26.456205 24947 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:26.559551 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.560253 24947 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:26.566706 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:26.527606 25098 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:26.528934 28633 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:26.569860 28633 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.571118 28633 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:26.537074 28394 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:26.538165 28412 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 13:20:26.539640 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54132.7us
I20240426 13:20:26.486819 25622 rpcz_store.cc:276] 0426 13:20:24.484645 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:24.560721 (+ 76076us) service_pool.cc:225] Handling call
0426 13:20:24.568885 (+  8164us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:24.569397 (+   512us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:24.569425 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:25.456860 (+887435us) log.cc:842] Serialized 52 byte log entry
0426 13:20:26.305942 (+849082us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:26.305997 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.306094 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.463045 (+156951us) raft_consensus.cc:1671] finished
0426 13:20:26.463072 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.463274 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2887,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":784919,"threads_started":1,"wal-append.queue_time_us":981775}
I20240426 13:20:26.501319 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:26.542081 28049 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.702s	user 0.009s	sys 0.002s
I20240426 13:20:26.542189 28560 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20240426 13:20:26.548223 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:26.527559 28423 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:20:26.547748 28630 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 13:20:26.517846 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:26.536176 28548 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:26.572257 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 156847us
I20240426 13:20:26.589721 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 635) took 2838 ms. Trace:
I20240426 13:20:26.589777 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:26.590760 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60081us
I20240426 13:20:26.593295 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.601449 28429 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:26.601997 28429 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:26.602974 28429 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.605284 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.584342 28394 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:26.606428 28394 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:26.593178 24951 rpcz_store.cc:276] 0426 13:20:23.751412 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.681003 (+1929591us) service_pool.cc:225] Handling call
0426 13:20:25.681359 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.681723 (+   364us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:26.374366 (+692643us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.588923 (+214557us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:26.588977 (+    54us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:26.589029 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.589530 (+   501us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.589666 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38630,"prepare.queue_time_us":217131,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":29846,"thread_start_us":816549,"threads_started":2}
I20240426 13:20:26.612222 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.613493 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:26.613348 28360 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:26.613760 28461 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:26.614835 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 642) took 2434 ms. Trace:
I20240426 13:20:26.622795 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:26.623111 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74950.1us
I20240426 13:20:26.624325 24947 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:26.624449 24954 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:26.628954 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 230866us
I20240426 13:20:26.633044 24956 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:26.626564 24948 rpcz_store.cc:276] 0426 13:20:24.180289 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.810728 (+1630439us) service_pool.cc:225] Handling call
0426 13:20:25.811098 (+   370us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.830396 (+ 19298us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:26.400546 (+570150us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.614473 (+213927us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:26.614535 (+    62us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:26.614594 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.614674 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.614787 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4954,"prepare.queue_time_us":58831,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":56,"thread_start_us":717147,"threads_started":2}
I20240426 13:20:26.638886 24955 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:26.589859 28049 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.643947 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.644603 28049 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.632470 28560 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:26.625191 28360 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:26.645857 28360 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:26.628155 28585 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:26.646431 28585 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:26.646775 28585 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:26.648111 28635 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:26.632833 28626 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:26.648945 28626 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:26.649871 28626 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.645570 28641 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
W20240426 13:20:26.655611 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 254579us
W20240426 13:20:26.657739 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101033us
I20240426 13:20:26.657054 24947 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 11. Ops: []
I20240426 13:20:26.628870 28660 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:26.663173 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2186) took 2158 ms. Trace:
I20240426 13:20:26.648362 28429 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:26.663641 24952 rpcz_store.cc:276] 0426 13:20:24.504492 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.828568 (+1324076us) service_pool.cc:225] Handling call
0426 13:20:25.838641 (+ 10073us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.854705 (+ 16064us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:26.397125 (+542420us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.662846 (+265721us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:26.662891 (+    45us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:26.662934 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.663002 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.663115 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25306,"prepare.queue_time_us":44688,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":746275,"threads_started":2}
I20240426 13:20:26.668494 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.613238 27911 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.670964 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.670804 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 633) took 2920 ms. Trace:
W20240426 13:20:26.675047 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43583us
I20240426 13:20:26.677616 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.666895 28661 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:26.678198 25102 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:26.673633 24953 rpcz_store.cc:276] 0426 13:20:23.750609 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.632340 (+1881731us) service_pool.cc:225] Handling call
0426 13:20:25.666698 (+ 34358us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:25.669633 (+  2935us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:26.372381 (+702748us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.670415 (+298034us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:26.670462 (+    47us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:26.670519 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.670575 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.670719 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55541,"prepare.queue_time_us":229066,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":80604,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":893633,"threads_started":3,"wal-append.queue_time_us":282049}
W20240426 13:20:26.721019 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47868.2us
I20240426 13:20:26.680876 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1617) took 1094 ms. Trace:
I20240426 13:20:26.675541 28660 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:26.732290 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:26.731935 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:26.722172 28394 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.738376 28360 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.740391 28360 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.678076 28653 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:26.672948 28062 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20240426 13:20:26.719271 28548 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.730118 28540 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:26.729934 25100 rpcz_store.cc:276] 0426 13:20:25.582503 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:25.587471 (+  4968us) service_pool.cc:225] Handling call
0426 13:20:25.631774 (+ 44303us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:25.643039 (+ 11265us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:25.643069 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:25.669131 (+ 26062us) log.cc:842] Serialized 26 byte log entry
0426 13:20:26.369204 (+700073us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:26.456855 (+ 87651us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b240024d898. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d247ce4 00007fda8d245dfb 00007fda8d2d3e09 00007fda8d2d3a95 00007fda8d2d30dd 00007fda8ff822ee 00007fda8ff99a8c 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1
0426 13:20:26.594552 (+137697us) log.cc:842] Serialized 14 byte log entry
0426 13:20:26.673294 (+ 78742us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:26.673365 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:26.673409 (+    44us) raft_consensus.cc:1671] finished
0426 13:20:26.676979 (+  3570us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:26.677181 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37924,"prepare.queue_time_us":171486,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":35,"raft.queue_time_us":97679,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":5570,"spinlock_wait_cycles":21245696,"thread_start_us":938913,"threads_started":3,"wal-append.queue_time_us":775272}
I20240426 13:20:26.741495 27911 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:26.777638 25102 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
W20240426 13:20:26.779886 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47677.2us
I20240426 13:20:26.779917 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.782850 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.769959 28540 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:26.786180 28540 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:26.785979 28660 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.766961 28653 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:26.789426 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.789520 28653 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.782060 28548 raft_consensus.cc:2741] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:20:26.793948 28671 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:26.795318 28671 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:26.795712 28671 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:26.795168 28540 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.797538 28540 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:26.783990 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.804749 24949 raft_consensus.cc:2396] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20240426 13:20:26.809558 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.810329 25029 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:26.810674 25102 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:26.811646 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.812932 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.815728 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.816797 25022 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:20:26.808774 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.817725 25023 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:26.818588 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.818815 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.819607 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.820590 25099 raft_consensus.cc:2354] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
W20240426 13:20:26.824851 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 165706us
I20240426 13:20:26.828632 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:26.831375 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.834499 25024 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:26.766925 28630 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20240426 13:20:26.747275 28279 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.840553 25025 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:26.747610 28642 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.818601 28540 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:26.842118 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:26.842447 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.848224 25028 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:26.846174 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.852687 25022 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:26.855931 25102 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240426 13:20:26.859359 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.861137 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:26.856324 25102 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 4
W20240426 13:20:26.846719 28662 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20240426 13:20:26.862605 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.863389 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:26.867719 25589 connection.cc:232] RPC call timeout handler was delayed by -2.3425605820000328s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20240426 13:20:26.868216 25589 connection.cc:232] RPC call timeout handler was delayed by -2.2273605820000739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20240426 13:20:26.870465 25589 connection.cc:232] RPC call timeout handler was delayed by -2.176060581999991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
W20240426 13:20:26.870885 25589 connection.cc:232] RPC call timeout handler was delayed by -2.1751605819999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
I20240426 13:20:26.862265 25102 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 7.5
W20240426 13:20:26.871240 25589 connection.cc:232] RPC call timeout handler was delayed by -2.1526605820000668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20240426 13:20:26.871523 25102 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:26.871666 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:26.847888 28394 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.869016 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:26.866520 25099 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:26.876147 25024 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 10 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20240426 13:20:26.878317 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:26.861312 28540 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:26.889494 25103 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:26.835351 28254 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:26.877995 28658 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20240426 13:20:26.895123 28279 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:26.896230 28642 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:26.896955 28254 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:26.901412 25099 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:26.901986 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.903390 25028 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:26.905680 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:26.907765 25029 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:26.908439 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.848114 28360 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20240426 13:20:26.846581 28661 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
W20240426 13:20:26.915428 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57706.4us
I20240426 13:20:26.873502 28062 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20240426 13:20:26.892115 28049 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:26.932281 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:26.935936 25622 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:26.930245 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:26.933399 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86515.7us
I20240426 13:20:26.937554 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:26.923549 28548 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:26.941988 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127492us
I20240426 13:20:26.942582 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:26.944844 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:26.945758 25098 raft_consensus.cc:595] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:26.948444 25099 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:26.948664 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.950940 25029 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:26.868207 28671 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:26.954524 25023 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:26.963105 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:27.037650 25029 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:26.969246 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:27.039687 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96578us
I20240426 13:20:26.942231 28670 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:27.031706 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63138.6us
I20240426 13:20:26.947942 28548 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:26.964025 28540 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:26.969448 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:27.036865 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:26.950307 28360 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:27.032480 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.051452 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:26.955015 28672 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:27.062539 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:27.039623 28630 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:27.039613 28049 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:27.057467 28548 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:27.077368 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 107988us
I20240426 13:20:27.074103 25023 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:27.082778 25099 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:27.086627 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.093297 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:27.083714 25027 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:27.095418 25027 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:27.095217 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:27.094113 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.068408 28394 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:27.099074 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:27.100852 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131707us
I20240426 13:20:27.077591 28642 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:27.079581 28630 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:27.124668 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.083897 28671 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:27.125922 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:27.130381 24921 leader_election.cc:398] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10.
I20240426 13:20:27.130765 24921 leader_election.cc:401] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20240426 13:20:27.131969 28633 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:27.141163 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.144018 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:27.147501 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60614.4us
I20240426 13:20:27.129246 28630 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:27.166265 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.131721 28440 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:27.125386 28437 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:27.129359 28394 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.136301 28580 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:27.128491 28360 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:27.168259 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.180552 25027 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:27.133010 28560 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:27.176118 25103 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:27.177234 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:27.188499 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:27.194376 25622 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:20:27.212110 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81902.1us
I20240426 13:20:27.177420 28437 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:27.177044 28440 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:27.179009 28580 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:27.222945 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:27.167694 28626 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240426 13:20:27.234688 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:27.209887 28360 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.225073 28437 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:27.224776 28440 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:27.266232 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96954.5us
W20240426 13:20:27.220574 28585 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.573s	user 0.018s	sys 0.010s
W20240426 13:20:27.237111 28688 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20240426 13:20:27.226145 28668 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:27.235630 28580 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:27.270897 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102724us
I20240426 13:20:27.176992 28049 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.275837 25096 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:27.274955 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:27.281272 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:27.263840 28680 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:27.298085 24951 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:27.271638 28585 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.273326 28062 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:27.304144 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:27.320544 28687 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20240426 13:20:27.327422 28708 raft_consensus.cc:2741] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
W20240426 13:20:27.379442 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75381.3us
W20240426 13:20:27.382141 28694 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20240426 13:20:27.346041 28652 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:27.382858 28692 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20240426 13:20:27.383028 28652 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:27.383973 28652 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:27.338901 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 121102us
W20240426 13:20:27.340679 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54842.1us
W20240426 13:20:27.344101 28429 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.676s	user 0.005s	sys 0.005s
I20240426 13:20:27.284768 27911 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
W20240426 13:20:27.325991 28690 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
W20240426 13:20:27.357101 28673 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
W20240426 13:20:27.358397 28683 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20240426 13:20:27.359462 28682 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
W20240426 13:20:27.360790 28684 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20240426 13:20:27.304023 28062 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:27.410486 28062 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:27.322073 28626 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:27.369129 28585 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:27.370261 28678 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20240426 13:20:27.328244 28680 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.272166 28394 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:27.373212 28698 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20240426 13:20:27.357631 28697 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:27.342154 25026 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:27.428064 28701 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20240426 13:20:27.429422 28701 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:27.429981 28701 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:27.431164 28701 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:27.420601 27911 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.398057 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:27.420130 28697 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:27.405476 25100 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:27.414346 25099 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:27.418588 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:27.344573 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121723us
I20240426 13:20:27.392844 25103 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:27.450228 28360 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:27.450773 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 167659us
I20240426 13:20:27.478942 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:27.482197 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 84267.2us
W20240426 13:20:27.482787 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5485751049999408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20240426 13:20:27.483354 24921 connection.cc:232] RPC call timeout handler was delayed by -1.543175104999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20240426 13:20:27.483750 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5098751049999919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20240426 13:20:27.484164 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3775751050000054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
W20240426 13:20:27.484556 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3532751049999661s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20240426 13:20:27.484910 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3190751050000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20240426 13:20:27.485208 24921 connection.cc:232] RPC call timeout handler was delayed by -1.2875751049999735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20240426 13:20:27.485508 24921 connection.cc:232] RPC call timeout handler was delayed by -1.2614751049999313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20240426 13:20:27.485811 24921 connection.cc:232] RPC call timeout handler was delayed by -1.2092751049999606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
W20240426 13:20:27.486102 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1930751049999344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20550000000000002s for a response
W20240426 13:20:27.486537 24921 connection.cc:232] RPC call timeout handler was delayed by -1.0463751049999246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
I20240426 13:20:27.489109 28711 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:27.489599 28711 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:27.489960 28711 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:27.492713 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74238.2us
I20240426 13:20:27.494917 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.495522 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.496132 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.496830 25031 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:27.496876 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.498073 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.499176 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.499519 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1191) took 1016 ms. Trace:
I20240426 13:20:27.499724 25025 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:27.496886 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 259590us
I20240426 13:20:27.499895 25105 rpcz_store.cc:276] 0426 13:20:26.483081 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.483252 (+   171us) service_pool.cc:225] Handling call
0426 13:20:27.499001 (+1015749us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:27.499446 (+   445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1818795776}
I20240426 13:20:27.500392 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.502027 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.503990 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.507479 25030 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:20:27.474238 28697 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:27.504818 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.508864 25031 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:27.515110 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:27.518252 25024 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:27.515390 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:27.482888 28659 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:27.522265 24950 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:27.487046 28702 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:27.527343 25104 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:27.527851 28689 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:27.536155 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:27.550149 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:27.550856 25023 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:27.526754 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:27.517473 28429 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:27.558327 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:27.559523 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 382341us
I20240426 13:20:27.541543 28560 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.578070 28585 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:27.544879 28652 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:27.597218 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:27.589691 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74875.2us
I20240426 13:20:27.590221 25025 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:20:27.596216 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78314.7us
I20240426 13:20:27.579113 27911 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.606793 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:27.607758 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.519905 28680 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.612015 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.653216 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.596952 28652 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:27.595957 28429 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.653996 25102 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:27.658149 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43733us
I20240426 13:20:27.672142 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:27.658949 28671 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.524s	user 0.005s	sys 0.005s
I20240426 13:20:27.607897 28585 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:27.660578 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:27.654330 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57223.6us
I20240426 13:20:27.586494 28437 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:27.659261 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61165.1us
I20240426 13:20:27.684520 25102 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:27.686247 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 97383.4us
I20240426 13:20:27.686404 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.687603 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:27.653383 28709 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:27.688832 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.688125 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.693425 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.668903 28652 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:27.695924 24995 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9.
I20240426 13:20:27.698074 24995 leader_election.cc:401] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240426 13:20:27.699383 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:27.687443 28719 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20240426 13:20:27.702620 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.611846 28560 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.710186 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:27.690188 28709 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:27.710925 25101 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:27.720225 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:27.673399 28580 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:27.726346 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:27.727955 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55962.6us
I20240426 13:20:27.707427 28668 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:27.738090 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.740257 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:27.738950 25102 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:27.747081 25101 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:27.724475 28709 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:27.756507 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:27.724606 28713 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:27.719437 28671 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20240426 13:20:27.748940 28708 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:27.766659 25102 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:27.769126 25101 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20240426 13:20:27.769548 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70245.1us
I20240426 13:20:27.770709 25102 pending_rounds.cc:76] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 2
I20240426 13:20:27.771090 25102 pending_rounds.cc:97] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:20:27.771489 25102 raft_consensus.cc:2866] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:27.757030 28714 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:27.785494 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:27.794119 28712 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20240426 13:20:27.793946 28062 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.748757 28429 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.769515 28708 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:27.748914 28626 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20240426 13:20:27.799356 25099 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20240426 13:20:27.792712 28633 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.660s	user 0.008s	sys 0.003s
I20240426 13:20:27.804081 25097 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:27.812551 28721 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:27.813071 28721 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:27.813444 28721 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:27.814298 28672 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:27.814972 28672 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:27.816159 28710 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.247s	user 0.001s	sys 0.001s
I20240426 13:20:27.816241 28716 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:27.816802 28716 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.817725 28717 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:27.818166 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61738.9us
I20240426 13:20:27.818523 28717 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:27.819406 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:27.820638 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2302) took 1044 ms. Trace:
I20240426 13:20:27.821146 25026 rpcz_store.cc:276] 0426 13:20:26.776515 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.780092 (+  3577us) service_pool.cc:225] Handling call
0426 13:20:27.820399 (+1040307us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:27.820568 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1217552768}
I20240426 13:20:27.817634 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 482) took 1034 ms. Trace:
I20240426 13:20:27.837998 25022 rpcz_store.cc:276] 0426 13:20:26.783582 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.925954 (+142372us) service_pool.cc:225] Handling call
0426 13:20:26.941597 (+ 15643us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:26.970251 (+ 28654us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:27.032572 (+ 62321us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:27.050555 (+ 17983us) log.cc:842] Serialized 26 byte log entry
0426 13:20:27.351765 (+301210us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:27.351821 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:27.351916 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:27.817338 (+465422us) raft_consensus.cc:1671] finished
0426 13:20:27.817365 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:27.817553 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78776,"raft.queue_time_us":8581,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"thread_start_us":258436,"threads_started":1,"wal-append.queue_time_us":362469}
I20240426 13:20:27.784453 28713 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240426 13:20:27.822410 28630 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.623s	user 0.004s	sys 0.007s
W20240426 13:20:27.881701 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 129621us
I20240426 13:20:27.806821 28708 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:27.876911 28412 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:27.802709 28706 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9.
I20240426 13:20:27.807685 28626 consensus_queue.cc:1225] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20240426 13:20:27.891800 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 162941us
W20240426 13:20:27.871335 28440 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.596s	user 0.010s	sys 0.005s
I20240426 13:20:27.841349 28062 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:27.846194 28717 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:27.874023 28548 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.777s	user 0.009s	sys 0.003s
I20240426 13:20:27.866391 28714 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.759928 28585 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:27.907912 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 122508us
I20240426 13:20:27.857367 28633 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10.
W20240426 13:20:27.900801 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 271915us
I20240426 13:20:27.888087 28412 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:27.808976 28715 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20240426 13:20:27.847803 28711 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.867981 28049 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20240426 13:20:27.897864 28732 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:27.900406 28437 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:27.950959 28580 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:27.898404 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 291690us
I20240426 13:20:27.932404 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:27.936118 28728 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:27.936903 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:27.937243 28726 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20240426 13:20:27.937680 28279 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:27.887794 28630 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.912667 28717 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:27.993302 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:27.898870 28440 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.817682 28716 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:27.905596 28548 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.001972 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.001989 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.042182 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:27.902016 28727 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:28.017424 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 170160us
I20240426 13:20:28.025364 25104 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:28.022425 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:28.029736 25099 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:20:28.030779 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:28.031617 25589 leader_election.cc:398] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:20:27.940604 28705 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.258s
W20240426 13:20:28.002740 24905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47944us
I20240426 13:20:28.023361 28732 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:28.071261 25101 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20240426 13:20:28.055673 24955 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20240426 13:20:28.059159 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:28.059979 25103 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:28.060014 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66831.3us
I20240426 13:20:28.073210 24947 raft_consensus.cc:2354] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12.
I20240426 13:20:28.073658 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:28.061159 25030 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:28.075454 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.064546 25097 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:28.028017 28279 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:28.069727 24950 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:28.078199 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49182.6us
I20240426 13:20:28.086345 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.080024 24947 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:28.071790 25101 pending_rounds.cc:76] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 7
I20240426 13:20:28.088426 25101 pending_rounds.cc:97] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20240426 13:20:28.088752 25101 raft_consensus.cc:2866] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:28.077181 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:28.095760 25616 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:28.098434 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:28.099349 24952 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:28.107713 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:28.108290 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:28.117861 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.068959 28728 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:28.190491 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:28.208065 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 132873us
I20240426 13:20:28.196318 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.210593 24955 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:28.211964 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
W20240426 13:20:28.213932 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109214us
I20240426 13:20:28.077490 28715 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:28.220916 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.224344 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 645) took 1039 ms. Trace:
W20240426 13:20:28.226918 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 140667us
I20240426 13:20:28.227952 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.093046 28716 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:28.220268 25101 pending_rounds.cc:76] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 6
I20240426 13:20:28.224750 25098 rpcz_store.cc:276] 0426 13:20:27.185102 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.185249 (+   147us) service_pool.cc:225] Handling call
0426 13:20:27.215742 (+ 30493us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:27.560550 (+344808us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:27.560608 (+    58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:27.588435 (+ 27827us) log.cc:842] Serialized 26 byte log entry
0426 13:20:27.705505 (+117070us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:27.705553 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:27.706355 (+   802us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:28.224122 (+517767us) raft_consensus.cc:1671] finished
0426 13:20:28.224143 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:28.224278 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":74975,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":27196,"thread_start_us":391375,"threads_started":2,"wal-append.queue_time_us":179042}
I20240426 13:20:28.231622 25101 pending_rounds.cc:97] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240426 13:20:28.232051 25101 raft_consensus.cc:2866] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:28.102105 28254 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:28.077872 28745 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:28.032598 28412 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:28.077706 28737 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:28.243211 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:28.101742 28408 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:28.246312 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:28.249446 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.221068 28715 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:28.217568 28672 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:28.254782 24948 raft_consensus.cc:2396] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20240426 13:20:28.263840 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.253005 28412 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:28.272707 25098 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:28.247762 28279 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.274662 28279 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:28.275460 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:28.268723 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 238629us
I20240426 13:20:28.235780 28254 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:28.025104 28714 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:28.276974 28254 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.278268 28254 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:28.278964 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.282771 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:28.238438 28437 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:28.294267 25025 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:28.248844 28408 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:28.250774 28723 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:28.249320 28737 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:28.297855 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.303100 28701 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:28.308290 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.309180 24948 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:28.310037 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.310592 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
W20240426 13:20:28.304051 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 634) took 1504 ms (client timeout 1999 ms). Trace:
I20240426 13:20:28.308128 24956 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:28.311023 25100 rpcz_store.cc:269] 0426 13:20:26.799681 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.799903 (+   222us) service_pool.cc:225] Handling call
0426 13:20:28.303843 (+1503940us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b540039e754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:28.303968 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332390144}
I20240426 13:20:28.321995 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.081077 28732 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:28.305116 28548 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:28.325881 24948 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:28.311173 28737 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:28.328377 28691 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20240426 13:20:28.328562 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:28.396041 24956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:28.328959 28663 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:28.399685 25616 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
W20240426 13:20:28.329303 28720 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20240426 13:20:28.330116 28744 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:28.402058 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:28.330899 28735 consensus_queue.cc:1159] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 7 status { last_received { term: 7 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:20:28.311477 28254 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20240426 13:20:28.332875 28743 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20240426 13:20:28.225771 28728 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.296278 28672 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:28.251186 28745 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.412789 25104 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:28.414213 24948 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:28.324855 28711 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:28.406895 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:28.438432 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:28.326185 28723 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
I20240426 13:20:28.485677 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:28.918594 24952 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.819s	user 0.011s	sys 0.000s
W20240426 13:20:28.534452 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.008s	sys 0.011s
W20240426 13:20:28.960669 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.008s	sys 0.032s
I20240426 13:20:28.443161 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:28.534868 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:28.961416 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.14197e+06us
W20240426 13:20:28.629520 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 347218us
I20240426 13:20:28.962302 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:28.961546 25023 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:28.959703 24952 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:28.538851 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 286402us
W20240426 13:20:28.972164 24956 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.000s
I20240426 13:20:28.972560 24956 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:28.972689 25022 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:28.974344 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:28.443454 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:28.529805 25100 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240426 13:20:28.980170 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 573304us
I20240426 13:20:28.544694 28740 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:28.971123 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:28.947162 28766 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:20:28.412371 28663 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 13:20:28.945416 28755 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20240426 13:20:28.605959 28708 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.708s	user 0.009s	sys 0.002s
I20240426 13:20:28.406486 28440 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:28.408239 28630 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:28.602455 28724 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20240426 13:20:29.052274 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.053284 24949 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:28.423496 28705 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:28.992136 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:29.061975 24956 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:28.993424 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:28.995908 28697 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.452s	user 0.009s	sys 0.001s
I20240426 13:20:28.996536 28740 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:29.069182 28740 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:28.421890 28701 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.583s
I20240426 13:20:29.063606 28254 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:29.073168 28254 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.073765 28254 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.009588 28758 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:29.020077 28745 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:29.024628 28748 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.632s	user 0.000s	sys 0.001s
I20240426 13:20:29.079052 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 493) took 1025 ms. Trace:
I20240426 13:20:29.026638 28663 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20240426 13:20:29.030036 28718 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.548s	user 0.005s	sys 0.006s
I20240426 13:20:29.042522 28711 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:29.042660 28721 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.229s	user 0.007s	sys 0.005s
I20240426 13:20:28.711828 28633 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:28.910966 28736 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.005s	sys 0.018s
W20240426 13:20:29.057436 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 228652us
I20240426 13:20:28.535882 28717 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:28.971652 28734 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.400s	user 0.001s	sys 0.000s
I20240426 13:20:29.067996 24948 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:28.912088 28733 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:28.482769 28714 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.585s
I20240426 13:20:28.411842 28744 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.403699 28715 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:28.491852 28408 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.376919 25618 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:29.413090 25022 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:29.413434 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1830) took 1103 ms. Trace:
I20240426 13:20:29.413839 24948 rpcz_store.cc:276] 0426 13:20:28.312566 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.993228 (+680662us) service_pool.cc:225] Handling call
0426 13:20:29.413377 (+420149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:29.415381 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:29.416091 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:29.371606 24949 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:29.420596 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1832) took 1015 ms. Trace:
I20240426 13:20:29.421013 24949 rpcz_store.cc:276] 0426 13:20:28.405015 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.969134 (+564119us) service_pool.cc:225] Handling call
0426 13:20:29.420545 (+451411us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:29.422091 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:29.317154 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 354924us
W20240426 13:20:29.368942 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 649) took 2800 ms (client timeout 2066 ms). Trace:
I20240426 13:20:29.424429 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:29.342082 28700 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.977s	user 0.000s	sys 0.018s
I20240426 13:20:29.241392 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:29.182906 28689 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.632s	user 0.009s	sys 0.003s
I20240426 13:20:29.437037 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:29.438246 24956 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.438822 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1828) took 1129 ms. Trace:
I20240426 13:20:29.439383 24956 rpcz_store.cc:276] 0426 13:20:28.309275 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.417894 (+1108619us) service_pool.cc:225] Handling call
0426 13:20:29.438754 (+ 20860us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:29.344856 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 500) took 1099 ms. Trace:
W20240426 13:20:29.115432 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.002s	sys 0.040s
W20240426 13:20:29.440568 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.997s	user 0.003s	sys 0.059s
I20240426 13:20:29.242121 28661 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:29.225996 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:29.204524 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 637) took 2363 ms (client timeout 1951 ms). Trace:
I20240426 13:20:29.133908 25624 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:29.106061 28768 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:29.316892 28652 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.608s	user 0.011s	sys 0.001s
W20240426 13:20:29.423875 24951 rpcz_store.cc:269] 0426 13:20:26.568731 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.613224 (+ 44493us) service_pool.cc:225] Handling call
0426 13:20:29.368712 (+2755488us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:29.368856 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1087140096}
I20240426 13:20:29.235102 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1643) took 1506 ms. Trace:
I20240426 13:20:29.187023 28670 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:29.411535 28736 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s	user 0.009s	sys 0.025s
I20240426 13:20:29.072670 28697 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:29.106331 28706 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.017s	sys 0.021s
I20240426 13:20:29.426575 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.434015 25617 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:29.107020 28691 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.351s
I20240426 13:20:29.434157 25023 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20240426 13:20:29.369146 28709 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.606s	user 0.005s	sys 0.007s
W20240426 13:20:29.342942 28716 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.099s	user 0.010s	sys 0.000s
I20240426 13:20:29.433096 24949 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:29.337931 28780 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20240426 13:20:29.186034 28279 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.911s	user 0.013s	sys 0.000s
W20240426 13:20:29.237524 28412 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.944s	user 0.008s	sys 0.001s
I20240426 13:20:29.377373 28702 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:29.441084 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 997373us
I20240426 13:20:29.528544 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:29.529070 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:29.419660 28721 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.130585 28705 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.078892 28440 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.259900 28732 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:29.440503 25616 rpcz_store.cc:276] 0426 13:20:28.245769 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.245987 (+   218us) service_pool.cc:225] Handling call
0426 13:20:29.344578 (+1098591us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:29.344785 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350138240}
I20240426 13:20:29.445063 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1827) took 1136 ms. Trace:
I20240426 13:20:29.550806 24947 rpcz_store.cc:276] 0426 13:20:28.308991 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.225643 (+916652us) service_pool.cc:225] Handling call
0426 13:20:29.444990 (+219347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:29.551872 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:29.445542 25096 rpcz_store.cc:269] 0426 13:20:26.845685 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.856568 (+ 10883us) service_pool.cc:225] Handling call
0426 13:20:29.204316 (+2347748us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:29.204456 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":927155968}
I20240426 13:20:29.385268 28728 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:29.134169 28254 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:29.158164 28741 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.332501 28779 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:29.456846 25102 rpcz_store.cc:276] 0426 13:20:27.728378 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.770093 (+ 41715us) service_pool.cc:225] Handling call
0426 13:20:27.770530 (+   437us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:27.787732 (+ 17202us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:27.787753 (+    21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:27.788192 (+   439us) log.cc:842] Serialized 26 byte log entry
0426 13:20:28.042777 (+254585us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:28.042816 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:28.042878 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:29.234799 (+1191921us) raft_consensus.cc:1671] finished
0426 13:20:29.234826 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.235028 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":145287,"threads_started":1,"wal-append.queue_time_us":379877}
I20240426 13:20:29.424225 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:29.576704 25028 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:29.079725 28585 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.376s
I20240426 13:20:29.465466 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.205029 28653 raft_consensus.cc:979] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:29.465235 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 490) took 2507 ms. Trace:
I20240426 13:20:29.097977 25026 rpcz_store.cc:276] 0426 13:20:28.053884 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.054272 (+   388us) service_pool.cc:225] Handling call
0426 13:20:28.078770 (+ 24498us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:28.078968 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:28.538607 (+459639us) log.cc:842] Serialized 14 byte log entry
0426 13:20:29.077960 (+539353us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:29.078362 (+   402us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:29.078601 (+   239us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:29.078862 (+   261us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.078991 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":461856,"prepare.queue_time_us":318287,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":49,"raft.queue_time_us":97913,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":50054,"thread_start_us":602524,"threads_started":2}
I20240426 13:20:29.581493 24951 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:29.488751 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.525096 28716 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:29.455732 28689 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 13:20:29.493189 25031 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.182s	user 0.008s	sys 0.026s
W20240426 13:20:29.596535 25031 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.286s	user 0.008s	sys 0.026s
I20240426 13:20:29.060336 28708 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.416767 28711 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.514313 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:29.460956 28642 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.418548 28733 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:29.610993 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96782.1us
I20240426 13:20:29.518127 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.612284 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:29.520102 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.007s	sys 0.023s
W20240426 13:20:29.613454 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.007s	sys 0.023s
I20240426 13:20:29.520818 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:29.435424 28785 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
W20240426 13:20:29.526993 25101 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.193s	user 0.015s	sys 0.029s
W20240426 13:20:29.501574 28706 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.966s	user 0.025s	sys 0.038s
W20240426 13:20:29.472051 28737 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.064s	user 0.010s	sys 0.001s
I20240426 13:20:29.624703 28737 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:29.625572 28750 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.406s	user 0.002s	sys 0.000s
I20240426 13:20:29.620540 28714 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20240426 13:20:29.627242 28737 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:29.627447 28789 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:20:29.615240 28686 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.628232 28729 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:29.628659 28729 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:29.628543 28686 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:29.629410 28758 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.553s	user 0.001s	sys 0.009s
I20240426 13:20:29.629933 28758 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:29.631206 28706 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 13:20:29.633989 28797 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.063s	user 0.002s	sys 0.000s
I20240426 13:20:29.608618 28653 raft_consensus.cc:1067] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:29.615602 28642 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.638774 28642 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:29.643100 24952 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.488698 28745 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.464560 28777 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
W20240426 13:20:29.539570 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.015s	sys 0.028s
I20240426 13:20:29.539680 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:29.465986 28670 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:29.520212 28784 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20240426 13:20:29.656116 28777 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:29.541224 25615 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:29.596805 28741 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.554765 25616 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:29.659425 28741 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:29.566301 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1996) took 1315 ms. Trace:
I20240426 13:20:29.659606 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:29.660286 25616 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:29.572073 24947 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.660769 28744 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:29.661417 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.572502 25617 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20240426 13:20:29.470602 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.549s (SENT) [suppressed 18 similar messages]
I20240426 13:20:29.465512 28652 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.581521 25027 rpcz_store.cc:276] 0426 13:20:26.958378 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.099416 (+141038us) service_pool.cc:225] Handling call
0426 13:20:27.099665 (+   249us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:27.180971 (+ 81306us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:29.464985 (+2284014us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b54003f7d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:29.465181 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185629056}
I20240426 13:20:29.593035 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:29.597409 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.540812 28707 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:29.543897 28626 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:29.524684 28062 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.057s
I20240426 13:20:29.549438 28630 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.568905 28540 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:29.515905 25029 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.000s	sys 0.028s
I20240426 13:20:29.612411 24951 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
W20240426 13:20:29.721644 25029 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.000s	sys 0.028s
I20240426 13:20:29.600003 28716 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.614321 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
W20240426 13:20:29.615871 25101 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.282s	user 0.015s	sys 0.029s
I20240426 13:20:29.728053 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1994) took 1674 ms. Trace:
I20240426 13:20:29.619609 25100 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:29.621095 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 650017us
I20240426 13:20:29.521581 28709 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.629019 28729 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:29.097991 25588 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.030s	sys 0.052s
W20240426 13:20:29.739476 25588 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 1.336s	user 0.039s	sys 0.072s
I20240426 13:20:29.728394 25101 rpcz_store.cc:276] 0426 13:20:28.053276 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.106418 (+ 53142us) service_pool.cc:225] Handling call
0426 13:20:28.219729 (+113311us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:28.292099 (+ 72370us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:28.292141 (+    42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:28.292735 (+   594us) log.cc:842] Serialized 52 byte log entry
0426 13:20:29.727681 (+1434946us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:29.727720 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:29.727793 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:29.727850 (+    57us) raft_consensus.cc:1671] finished
0426 13:20:29.727861 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.727989 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36058,"raft.queue_time_us":114,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":5931,"thread_start_us":1393713,"threads_started":1,"wal-append.queue_time_us":1316668}
I20240426 13:20:29.536620 24953 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:29.741691 24953 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:29.741879 24953 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:20:29.742043 24953 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:29.644012 25104 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:29.651037 25620 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:29.465571 28768 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:29.654002 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.016s	sys 0.028s
W20240426 13:20:29.747371 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 754420us
I20240426 13:20:29.653374 28745 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.727130 28737 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.760304 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:29.599041 28779 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.657120 28777 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.536052 28702 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:29.579412 25622 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:29.767771 25098 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:29.566582 28440 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.659999 25097 rpcz_store.cc:276] 0426 13:20:28.250443 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.250628 (+   185us) service_pool.cc:225] Handling call
0426 13:20:28.271231 (+ 20603us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:28.291177 (+ 19946us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:28.291208 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:28.291618 (+   410us) log.cc:842] Serialized 26 byte log entry
0426 13:20:28.564618 (+273000us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:28.564675 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:28.564774 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:29.337190 (+772416us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:29.565917 (+228727us) raft_consensus.cc:1671] finished
0426 13:20:29.565949 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.566227 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19982,"spinlock_wait_cycles":30203520,"thread_start_us":231040,"threads_started":1,"wal-append.queue_time_us":928838}
I20240426 13:20:29.607954 28254 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:29.612444 28697 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:29.682103 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:29.774354 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 92358.6us
I20240426 13:20:29.675294 24948 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.622059 28728 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:29.694532 25025 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:29.622581 28279 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.696573 28758 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:29.697127 28711 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:29.699283 28731 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:29.838017 28731 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:29.839051 28731 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.714910 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:29.722411 24956 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:29.722453 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.841506 24951 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:29.491799 28786 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.126s
I20240426 13:20:29.722379 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 495) took 1659 ms. Trace:
I20240426 13:20:29.549046 28705 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.731521 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:29.726141 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2325) took 1427 ms. Trace:
W20240426 13:20:29.725979 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 116172us
I20240426 13:20:29.734006 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:29.736702 28709 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:29.737092 28717 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:29.738293 24947 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:29.752645 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 137073us
I20240426 13:20:29.622315 28787 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.034s
W20240426 13:20:29.770738 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 298755us
I20240426 13:20:29.771916 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:29.775221 25615 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:29.776738 28793 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:29.777086 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.777621 28736 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:29.916431 24948 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:29.778498 28802 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20240426 13:20:29.779930 28803 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20240426 13:20:29.790264 28728 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:29.726495 28630 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.790999 28795 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:29.791563 28794 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:29.791954 28653 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:29.738839 28540 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.768492 28737 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.775651 28254 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:29.785531 28686 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:29.722518 28707 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:29.782393 28652 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:29.722575 28626 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:29.829452 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 88187.2us
I20240426 13:20:29.836025 24949 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20240426 13:20:29.781315 28745 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:29.855722 25031 rpcz_store.cc:276] 0426 13:20:28.298361 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.298515 (+   154us) service_pool.cc:225] Handling call
0426 13:20:28.310324 (+ 11809us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:28.310531 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:29.596937 (+1286406us) log.cc:842] Serialized 14 byte log entry
0426 13:20:29.712628 (+115691us) log.cc:842] Serialized 14 byte log entry
0426 13:20:29.725810 (+ 13182us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:29.725866 (+    56us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:29.725919 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:29.725987 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.726094 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":649954,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":69,"raft.queue_time_us":15654,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":12906,"spinlock_wait_cycles":52630144,"thread_start_us":1400999,"threads_started":2,"wal-append.queue_time_us":175151}
W20240426 13:20:29.862416 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 111964us
I20240426 13:20:29.876555 28661 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:29.877727 28716 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:29.916323 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
W20240426 13:20:29.918345 24995 leader_election.cc:334] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.578s (SENT)
I20240426 13:20:29.918473 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:30.032006 25619 pending_rounds.cc:76] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 10
I20240426 13:20:30.032251 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:30.032773 25619 pending_rounds.cc:97] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20240426 13:20:30.033463 25619 raft_consensus.cc:2866] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:29.852896 25029 rpcz_store.cc:276] 0426 13:20:28.062541 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.062711 (+   170us) service_pool.cc:225] Handling call
0426 13:20:28.062924 (+   213us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:28.078233 (+ 15309us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:28.321922 (+243689us) log.cc:842] Serialized 14 byte log entry
0426 13:20:29.722059 (+1400137us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:29.722110 (+    51us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:29.722162 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:29.722225 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:29.722331 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":494644,"prepare.queue_time_us":317657,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":45128,"raft.queue_time_us":180510,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":9045,"thread_start_us":1325505,"threads_started":3,"wal-append.queue_time_us":869530}
I20240426 13:20:29.867035 24951 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:29.928851 25617 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:30.031795 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 161681us
I20240426 13:20:29.955983 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:30.036298 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:30.046432 24948 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:30.041154 25624 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:30.047839 25618 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
W20240426 13:20:30.035442 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 303965us
I20240426 13:20:30.051443 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:29.915808 28711 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:30.086745 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 168436us
I20240426 13:20:30.030853 28809 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.098279 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.032006 28653 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:30.104550 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:29.844736 28279 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.928563 28736 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:30.129812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:29.966114 28705 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.042407 28728 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:29.928357 28793 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.119644 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.119757 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.144243 25028 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:30.145169 28793 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.145344 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:30.146149 25100 raft_consensus.cc:2354] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:20:30.120467 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.148540 25101 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:30.148842 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.149468 25023 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:30.121670 28709 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.150489 28709 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.152263 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:30.121148 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.156672 25028 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:30.122936 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:30.128311 24995 leader_election.cc:334] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.626s (SENT)
W20240426 13:20:30.107640 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72307.6us
I20240426 13:20:30.158036 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:30.064579 28808 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:30.064921 28652 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:30.064724 28745 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.137998 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.141031 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.153102 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:30.128624 28653 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:30.164641 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1993) took 2432 ms. Trace:
I20240426 13:20:30.165952 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:30.163623 28793 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:30.076864 28794 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.175050 28794 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.141939 28708 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.118953 28711 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.136142 28733 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.133445 28736 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:30.181857 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:30.182686 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61613us
I20240426 13:20:30.169824 25099 rpcz_store.cc:276] 0426 13:20:27.731868 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.805710 (+ 73842us) service_pool.cc:225] Handling call
0426 13:20:27.805912 (+   202us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:28.057813 (+251901us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:30.164401 (+2106588us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:30.164588 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514184960}
I20240426 13:20:30.198688 25099 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20240426 13:20:30.163142 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:30.161345 28652 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:30.337577 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:30.354795 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:30.212945 28686 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:30.168790 28745 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.170500 25617 raft_consensus.cc:2396] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240426 13:20:30.188475 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:29.986323 28702 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:29.966595 28670 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.137890 28440 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.194531 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:30.114703 28787 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20240426 13:20:30.184702 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:30.396546 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 212498us
I20240426 13:20:30.168079 28062 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:30.217058 28652 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:30.130489 28809 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.190752 25028 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:30.223291 28708 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.173s
I20240426 13:20:30.386618 28794 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:30.426149 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:30.445082 25097 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:30.429821 28681 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:30.437815 28745 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.439774 28670 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.440840 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:30.441366 28809 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.396560 28733 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:30.448441 28681 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:30.408293 28062 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:30.417485 28440 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:30.420998 28787 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:30.420677 28786 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:30.409915 28686 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.444512 28736 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.525617 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:30.642863 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.065s	sys 0.139s
I20240426 13:20:30.675081 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:30.675561 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.066s	sys 0.140s
W20240426 13:20:30.555625 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 427311us
W20240426 13:20:30.555914 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 201203us
I20240426 13:20:30.679098 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.484333 28733 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:30.680343 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:30.682925 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.683506 25024 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:30.488981 28254 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:30.517221 28729 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.756s	user 0.010s	sys 0.000s
I20240426 13:20:30.676610 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.490727 28062 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:30.693935 24995 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.642s (SENT)
I20240426 13:20:30.526307 28707 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:30.516814 28630 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.696642 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.697057 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.697463 25022 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:30.697618 25024 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:30.696813 28630 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:30.700981 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.704742 28817 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.705310 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:30.705255 28817 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.706204 28729 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.707406 28817 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.708235 28729 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:30.709704 28795 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.633s	user 0.009s	sys 0.000s
I20240426 13:20:30.710142 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.710942 25024 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:30.712265 25618 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3014c46944bc4cbb82624ac909221634"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
W20240426 13:20:30.712548 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1634) took 3052 ms (client timeout 3023 ms). Trace:
I20240426 13:20:30.712878 25618 raft_consensus.cc:595] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:30.713600 25618 raft_consensus.cc:603] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:20:30.714145 28820 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20240426 13:20:30.712847 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.715065 25030 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:30.715435 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.715602 28670 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.716121 28670 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:30.716589 28763 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.875s	user 0.009s	sys 0.024s
I20240426 13:20:30.718187 28670 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:30.713048 25103 rpcz_store.cc:269] 0426 13:20:27.660008 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.662625 (+  2617us) service_pool.cc:225] Handling call
0426 13:20:30.712349 (+3049724us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54003b0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:30.712487 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":845596544}
I20240426 13:20:30.719693 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2007) took 1125 ms. Trace:
I20240426 13:20:30.685427 28758 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:30.490916 28681 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:30.720917 25098 rpcz_store.cc:276] 0426 13:20:29.594791 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.594917 (+   126us) service_pool.cc:225] Handling call
0426 13:20:29.616337 (+ 21420us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.768355 (+152018us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:30.719455 (+951100us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:30.719636 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347348224}
I20240426 13:20:30.723371 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:30.490757 28794 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:30.724946 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:30.695983 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.599720 28279 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.725889 25103 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:30.560405 28702 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.515941 28711 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.680188 28540 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:30.735736 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 327796us
I20240426 13:20:30.686270 28733 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:30.730912 25101 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:20:30.731736 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.741559 25030 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:30.689224 28708 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:30.710135 28795 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:30.770871 25616 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:30.726835 28823 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:30.733103 28758 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:30.773054 28758 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.775077 28758 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.727792 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.707912 28817 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:30.597846 28653 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.775979 28817 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.737152 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:30.777215 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 501) took 2382 ms (client timeout 2006 ms). Trace:
I20240426 13:20:30.737823 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:30.777758 25615 rpcz_store.cc:269] 0426 13:20:28.400218 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.408365 (+  8147us) service_pool.cc:225] Handling call
0426 13:20:30.776981 (+2368616us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54005c1054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:30.777120 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1339174656}
I20240426 13:20:30.741919 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.755034 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.756614 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:30.760630 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.761356 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.760793 28794 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.762180 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.515973 28440 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:30.769893 28728 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.605s	user 0.009s	sys 0.000s
I20240426 13:20:30.727653 28707 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:30.777828 28817 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.779083 25024 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:20:30.819443 24920 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items.
I20240426 13:20:30.780213 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 504) took 2364 ms. Trace:
W20240426 13:20:30.820387 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126474us
I20240426 13:20:30.820770 25619 rpcz_store.cc:276] 0426 13:20:28.415388 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.415509 (+   121us) service_pool.cc:225] Handling call
0426 13:20:29.347746 (+932237us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:30.034238 (+686492us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:30.779115 (+744877us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:30.779960 (+   845us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54005a2dd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:30.780130 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3110099712}
I20240426 13:20:30.780380 28652 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.781433 25098 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:30.822753 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.823062 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.775219 25030 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:30.556823 28787 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.785307 28823 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:30.787066 25623 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:30.788286 28633 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:30.791118 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1838) took 1548 ms (client timeout 1892 ms). Trace:
W20240426 13:20:30.884608 24947 rpcz_store.cc:269] 0426 13:20:29.242429 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.661220 (+418791us) service_pool.cc:225] Handling call
0426 13:20:30.790945 (+1129725us) spinlock_profiling.cc:243] Waited 76.1 ms on lock 0x7b54001d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:30.791055 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174995712}
I20240426 13:20:30.792980 28641 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:30.793344 28758 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.874382 28767 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:30.795221 28711 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.796993 28768 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.561600 28709 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.813536 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.815822 25103 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:30.818150 28827 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:30.821772 24995 leader_election.cc:334] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.721s (SENT)
I20240426 13:20:30.824637 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.902283 25030 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:30.836374 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:30.903173 25031 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:30.837349 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:30.839540 28652 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.846340 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.907792 25098 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:30.823374 28744 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 13:20:30.851519 28560 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:30.853122 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1831) took 2458 ms (client timeout 2652 ms). Trace:
W20240426 13:20:30.911252 24956 rpcz_store.cc:269] 0426 13:20:28.394618 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.466491 (+1071873us) service_pool.cc:225] Handling call
0426 13:20:30.852873 (+1386382us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000a73d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:30.853049 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":252269824}
I20240426 13:20:30.889276 28633 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:30.867720 25621 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:30.872629 28668 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:30.881433 25105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.030s	sys 0.054s
I20240426 13:20:30.818652 28728 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.777172 28653 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.794373 28824 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.783108 28733 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.861377 28680 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:30.916585 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94824.5us
W20240426 13:20:30.919190 24921 connection.cc:232] RPC call timeout handler was delayed by -1.9040519000000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20240426 13:20:30.937877 24921 connection.cc:232] RPC call timeout handler was delayed by -1.7348518999999669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20240426 13:20:30.938406 24921 connection.cc:232] RPC call timeout handler was delayed by -1.713251899999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20240426 13:20:30.938810 24921 connection.cc:232] RPC call timeout handler was delayed by -1.713251899999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20240426 13:20:30.939172 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5584519000000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
I20240426 13:20:30.895543 28767 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:30.881538 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:30.921530 25621 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:30.910650 28670 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.936582 28668 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:30.941608 28680 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:30.939654 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5053518999999369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1978s for a response
W20240426 13:20:30.947502 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4342518999999356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20240426 13:20:30.947824 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3514519000000291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20240426 13:20:30.948132 24921 connection.cc:232] RPC call timeout handler was delayed by -1.313651899999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20240426 13:20:30.934614 25105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.030s	sys 0.054s
I20240426 13:20:30.948678 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.948616 28680 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.950353 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.890697 28641 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:30.945209 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.936807 28631 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:30.941445 28824 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.946215 28733 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.946676 28670 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.952617 28787 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:30.952418 28631 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.954053 28670 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:30.951843 28641 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.945531 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.904806 28708 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:30.913471 28768 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:30.957890 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.955265 28641 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:30.957139 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.972088 25101 raft_consensus.cc:2354] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:30.974874 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:30.971979 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.969635 28733 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:30.979729 28733 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:30.898725 28758 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:30.967515 25621 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.879760 24920 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2210)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 674)
I20240426 13:20:30.984212 25621 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:30.986053 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:30.986863 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.971560 28279 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:30.954053 28824 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:30.996919 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:30.913003 28711 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.981766 28793 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.003659 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:30.933132 28560 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.015667 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.013861 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:31.016562 25100 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:31.020887 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.026422 25589 leader_election.cc:398] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:20:31.018738 25031 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:31.026741 25589 leader_election.cc:401] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240426 13:20:30.918457 28827 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:30.999768 28686 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.034859 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.042673 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.042997 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.043759 25023 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:31.044996 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:31.045591 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48817.3us
I20240426 13:20:31.046533 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:30.981372 28768 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:31.050087 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 230160us
I20240426 13:20:31.050745 25103 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:31.041146 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:31.026605 28639 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:31.061759 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.062911 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:31.008632 28816 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:31.062062 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.027326 28744 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:31.053009 28669 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:31.067538 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.065172 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.070297 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.071638 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:31.069516 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.067027 25031 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:31.127772 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:30.993956 28659 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20240426 13:20:31.128800 25023 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:31.049158 28762 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:31.042493 28815 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:20:31.061198 28663 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:31.064097 28826 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:31.145855 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.148524 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:31.150501 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102691us
I20240426 13:20:31.161378 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.066236 28631 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:31.126865 28793 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:31.127708 28440 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:31.129638 28639 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:31.136152 28686 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.130033 28823 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.164345 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.143599 28663 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:31.177903 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146194us
I20240426 13:20:31.163273 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:31.182389 25619 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:31.171345 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.145601 28670 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:31.182932 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:31.183641 25617 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:31.137035 28279 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:31.189464 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:31.050284 28708 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.192639 25029 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:31.139194 28733 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20240426 13:20:31.174342 28826 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:31.195657 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.178262 28728 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:31.181552 28663 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:31.032294 28767 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.223989 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.239143 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:31.240741 28836 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20240426 13:20:31.240620 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.241755 25022 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:31.224668 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.242749 25031 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:31.243592 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 716) took 1529 ms (client timeout 1816 ms). Trace:
W20240426 13:20:31.244022 25616 rpcz_store.cc:269] 0426 13:20:29.714404 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.714578 (+   174us) service_pool.cc:225] Handling call
0426 13:20:31.243399 (+1528821us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.243533 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2137857408}
I20240426 13:20:31.248395 28758 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:31.225627 25026 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:31.231644 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.233565 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.191026 28793 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:31.184484 28846 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:31.191151 28740 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:31.237540 28838 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20240426 13:20:31.194808 28670 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:31.257002 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 513) took 1338 ms. Trace:
W20240426 13:20:31.285728 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 96345.7us
I20240426 13:20:31.182062 28686 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.259987 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.290843 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.290222 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:31.261165 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.185822 28823 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.254774 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:31.256372 28731 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:31.293970 25023 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:31.294528 28823 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.288214 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:31.275180 28062 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.000s
W20240426 13:20:31.295878 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 140800us
I20240426 13:20:31.282833 25624 rpcz_store.cc:276] 0426 13:20:29.918656 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.066830 (+148174us) service_pool.cc:225] Handling call
0426 13:20:30.067378 (+   548us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:30.198448 (+131070us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:30.198480 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:30.206783 (+  8303us) log.cc:842] Serialized 26 byte log entry
0426 13:20:30.898136 (+691353us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:31.126941 (+228805us) log.cc:842] Serialized 14 byte log entry
0426 13:20:31.155494 (+ 28553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.155591 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:31.256818 (+101227us) raft_consensus.cc:1671] finished
0426 13:20:31.256847 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:31.256958 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12620,"prepare.queue_time_us":291676,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":3672,"raft.queue_time_us":113571,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":23103,"thread_start_us":679805,"threads_started":3,"wal-append.queue_time_us":694845}
I20240426 13:20:31.270387 28767 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:31.270987 28631 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.257586 28793 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:31.297477 25026 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:31.159518 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 674)
I20240426 13:20:31.298476 25100 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:31.274662 28728 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:31.259160 28279 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:31.271853 28711 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.406499 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.407171 25100 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:31.408068 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.406174 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2239)
I20240426 13:20:31.292004 28733 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.412513 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.414749 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.296092 28762 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.296924 28846 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.421372 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:31.424130 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.423426 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.384337 28793 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:31.367190 28560 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.256665 28668 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.402211 28728 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:31.289848 28740 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:31.455940 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 168200us
I20240426 13:20:31.463690 25621 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20240426 13:20:31.466722 25621 pending_rounds.cc:76] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 10
I20240426 13:20:31.467072 25621 pending_rounds.cc:97] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20240426 13:20:31.467442 25621 raft_consensus.cc:2866] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:31.468411 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 505) took 2985 ms. Trace:
I20240426 13:20:31.468878 25621 rpcz_store.cc:276] 0426 13:20:28.483082 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.538674 (+ 55592us) service_pool.cc:225] Handling call
0426 13:20:28.879035 (+340361us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:30.867589 (+1988554us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003816a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:20:31.468066 (+600477us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.468346 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":662483200}
I20240426 13:20:31.416851 28631 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:31.439353 28768 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.367657 28062 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:31.493191 25069 leader_election.cc:334] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.750s (SENT)
I20240426 13:20:31.470023 28728 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:31.495926 28773 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20240426 13:20:31.470641 28670 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.445039 28762 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.496706 28827 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.498042 28850 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
W20240426 13:20:31.528239 28834 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.357s	user 0.001s	sys 0.000s
I20240426 13:20:31.528748 28850 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
W20240426 13:20:31.522789 28840 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.332s	user 0.000s	sys 0.000s
I20240426 13:20:31.496973 28279 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:31.522171 28841 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.286s	user 0.000s	sys 0.000s
I20240426 13:20:31.529472 28762 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:31.532688 25099 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:31.535460 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:20:31.521420 28848 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20240426 13:20:31.529191 28850 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:31.497066 28560 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.560299 28850 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.561096 28560 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.561641 28850 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:31.537619 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:31.458425 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41286 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2234)
I20240426 13:20:31.273921 28639 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.542766 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.462584 28844 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:31.561405 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.543888 25100 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:31.570006 25100 raft_consensus.cc:603] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:31.563740 25618 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:31.563211 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:31.566402 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276310us
I20240426 13:20:31.562969 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:31.501237 28839 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.243s	user 0.000s	sys 0.000s
W20240426 13:20:31.575187 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82037.8us
W20240426 13:20:31.576308 25069 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.895s (SENT)
I20240426 13:20:31.576828 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.573428 25028 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:31.577965 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.581480 24919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2234) took 1379 ms. Trace:
I20240426 13:20:31.569818 28844 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.582540 28408 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:31.582731 28560 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.582559 28765 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:31.583357 28828 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:31.580138 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.589262 28668 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.593652 25619 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:31.562234 28631 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:31.602329 28686 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.602211 28631 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.599849 28846 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:31.603849 28826 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:31.596357 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.581676 28670 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:31.596879 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.578497 28756 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:31.599167 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:31.599450 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.580829 28794 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:31.605422 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.594925 24919 rpcz_store.cc:276] 0426 13:20:30.202463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.581435 (+1378972us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:31.617534 25621 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:31.595721 28823 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:31.595616 28844 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:31.620941 28828 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:31.644884 28741 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:31.645177 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 483878us
I20240426 13:20:31.647346 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.621404 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:31.631687 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.649996 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.651444 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:31.618086 24919 service_pool.cc:138] RunLeaderElection request on kudu.consensus.ConsensusService from 127.0.0.1:41286 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233)
I20240426 13:20:31.660476 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:31.627553 28633 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.687s	user 0.007s	sys 0.004s
I20240426 13:20:31.622846 28756 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:31.666873 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.622584 28062 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.671546 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.685083 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.691774 25026 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:31.674048 24919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2233) took 1507 ms. Trace:
I20240426 13:20:31.692462 24919 rpcz_store.cc:276] 0426 13:20:30.166622 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.673982 (+1507360us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:31.686589 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.644024 28707 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.684561 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.695976 25028 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:31.699643 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.701565 25029 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:31.722074 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.686115 28360 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:31.661655 28852 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:31.686523 28746 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:31.628907 28793 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.667209 28756 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.680512 28798 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:31.663012 28828 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.662662 28823 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.685019 28663 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.686424 28633 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.745803 28633 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.730619 28827 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
W20240426 13:20:31.734935 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 88977us
I20240426 13:20:31.748425 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.750036 24995 leader_election.cc:398] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:31.750687 28707 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:31.751224 28707 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:31.750337 24995 leader_election.cc:401] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:31.653961 28826 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:31.712519 25620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 496) took 3686 ms (client timeout 2115 ms). Trace:
W20240426 13:20:31.772313 25620 rpcz_store.cc:269] 0426 13:20:28.025623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.025778 (+   155us) service_pool.cc:225] Handling call
0426 13:20:31.712350 (+3686572us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.712452 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3578378496}
I20240426 13:20:31.715031 28714 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.708695 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41286 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671)
I20240426 13:20:31.662098 28846 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:31.715801 28845 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.424s	user 0.001s	sys 0.010s
W20240426 13:20:31.716109 28777 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.330s	user 0.016s	sys 0.038s
I20240426 13:20:31.665455 28686 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:31.688606 28659 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.724640 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:31.694196 28794 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:31.788496 28631 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:31.789203 28631 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:31.790333 28631 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.777676 28714 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.796161 28794 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:31.797403 28794 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.799618 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1651) took 1058 ms. Trace:
I20240426 13:20:31.630892 28670 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.728132 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.728684 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2001) took 2192 ms. Trace:
I20240426 13:20:31.717166 28809 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:31.708506 28740 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:31.730077 28861 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20240426 13:20:31.806988 28681 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.077s	user 0.010s	sys 0.001s
I20240426 13:20:31.726522 28852 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.807430 28681 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:31.718549 28731 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:31.810961 25096 raft_consensus.cc:2354] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:20:31.718832 28672 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.719167 28062 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:31.728116 28756 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:31.728183 28798 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:31.741278 28793 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.741461 28823 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.828405 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2009) took 1739 ms. Trace:
I20240426 13:20:31.828899 25097 rpcz_store.cc:276] 0426 13:20:30.088671 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.088819 (+   148us) service_pool.cc:225] Handling call
0426 13:20:30.119818 (+ 30999us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:30.445630 (+325812us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.828110 (+1382480us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:31.828333 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":697988352}
I20240426 13:20:31.829659 28844 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.830552 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.831725 28844 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:31.728191 28360 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:31.844383 28788 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.456s	user 0.014s	sys 0.008s
I20240426 13:20:31.845827 28360 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:31.753262 25100 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:31.848407 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1824) took 3543 ms (client timeout 1893 ms). Trace:
I20240426 13:20:31.848620 28809 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:31.848930 24952 rpcz_store.cc:269] 0426 13:20:28.305212 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.974076 (+668864us) service_pool.cc:225] Handling call
0426 13:20:31.848159 (+2874083us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.848329 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1534637440}
W20240426 13:20:31.850270 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2230) took 2099 ms (client timeout 1881 ms). Trace:
W20240426 13:20:31.850733 24952 rpcz_store.cc:269] 0426 13:20:29.753612 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.850036 (+2096424us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:31.850174 (+   138us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:31.851622 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2225) took 2194 ms (client timeout 2036 ms). Trace:
I20240426 13:20:31.852097 28756 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.854221 28756 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:31.854620 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1847) took 1818 ms (client timeout 1657 ms). Trace:
W20240426 13:20:31.855113 24955 rpcz_store.cc:269] 0426 13:20:30.036172 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.854411 (+1818239us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:31.854548 (+   137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:31.856012 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2229) took 2122 ms (client timeout 2035 ms). Trace:
W20240426 13:20:31.856462 24955 rpcz_store.cc:269] 0426 13:20:29.734092 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.855825 (+2121733us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:31.855948 (+   123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:31.857375 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2222) took 2201 ms (client timeout 2156 ms). Trace:
W20240426 13:20:31.857762 24955 rpcz_store.cc:269] 0426 13:20:29.656057 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.857187 (+2201130us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:31.857318 (+   131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:31.858826 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:31.871349 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.622493 28855 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:31.876392 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1649) took 1142 ms. Trace:
I20240426 13:20:31.759888 25029 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:31.877447 25029 pending_rounds.cc:76] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:20:31.877851 25029 pending_rounds.cc:97] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:20:31.878273 25029 raft_consensus.cc:2866] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:31.783320 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58744.9us
W20240426 13:20:31.859828 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1835) took 2812 ms (client timeout 2939 ms). Trace:
W20240426 13:20:31.782346 28777 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.442s	user 0.016s	sys 0.038s
I20240426 13:20:31.752136 28707 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:31.782557 28686 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.784483 28659 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.787029 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1684) took 1032 ms. Trace:
I20240426 13:20:31.800191 25103 rpcz_store.cc:276] 0426 13:20:30.741140 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.918630 (+177490us) service_pool.cc:225] Handling call
0426 13:20:31.799411 (+880781us) spinlock_profiling.cc:243] Waited 98.3 ms on lock 0x7b54003ef8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.799547 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":226182912}
W20240426 13:20:31.801069 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 140697us
I20240426 13:20:31.802933 24948 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:31.805467 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55112.5us
I20240426 13:20:31.897590 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:31.775655 28846 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.807888 28852 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:31.726393 28746 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:31.808689 24919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 671) took 1647 ms. Trace:
W20240426 13:20:31.783546 28641 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.802s	user 0.011s	sys 0.001s
I20240426 13:20:31.812230 25616 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:31.817533 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.803063 28706 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:31.804170 25105 rpcz_store.cc:276] 0426 13:20:29.536504 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.536697 (+   193us) service_pool.cc:225] Handling call
0426 13:20:29.556454 (+ 19757us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:30.115528 (+559074us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:30.115571 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:30.132616 (+ 17045us) log.cc:842] Serialized 26 byte log entry
0426 13:20:30.949002 (+816386us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:30.949041 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.722813 (+773772us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:31.722889 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:31.722967 (+    78us) raft_consensus.cc:1671] finished
0426 13:20:31.722986 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:31.728629 (+  5643us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46775,"raft.queue_time_us":47436,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":19734,"spinlock_wait_cycles":1177025792,"thread_start_us":767486,"threads_started":1,"wal-append.queue_time_us":859182}
I20240426 13:20:31.822651 28721 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:31.823575 28062 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:31.824404 28798 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.815826 28731 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.826683 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1818) took 3595 ms. Trace:
I20240426 13:20:31.898352 28717 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:31.743252 28663 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:31.841686 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2211) took 2359 ms (client timeout 1521 ms). Trace:
W20240426 13:20:31.925426 24949 rpcz_store.cc:269] 0426 13:20:29.481904 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.511969 (+ 30065us) service_pool.cc:225] Handling call
0426 13:20:31.841421 (+2329452us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.841602 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679848704}
I20240426 13:20:31.763062 25028 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:31.878691 25101 rpcz_store.cc:276] 0426 13:20:30.734209 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.756363 (+ 22154us) service_pool.cc:225] Handling call
0426 13:20:31.876146 (+1119783us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540001bdd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:31.876323 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369498880}
I20240426 13:20:31.831912 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:31.928715 25097 raft_consensus.cc:595] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:31.880841 24955 rpcz_store.cc:269] 0426 13:20:29.048403 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.858476 (+2810073us) service_pool.cc:225] Handling call
0426 13:20:31.859766 (+  1290us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:31.879343 24952 rpcz_store.cc:269] 0426 13:20:29.656827 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.851441 (+2194614us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:31.851565 (+   124us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:31.887081 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.893754 28722 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:31.898806 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.946938 28737 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:31.947835 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49122.6us
I20240426 13:20:31.948267 28706 raft_consensus.cc:2741] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20240426 13:20:31.949322 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:31.907788 28870 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:31.950425 28870 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.950786 28870 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:31.951378 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.953648 28641 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:31.954150 28872 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.117s	user 0.002s	sys 0.000s
I20240426 13:20:31.954336 28641 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:31.954804 28641 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:31.956460 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:31.955683 28641 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:31.957496 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 670) took 1798 ms (client timeout 2062 ms). Trace:
W20240426 13:20:31.957999 24952 rpcz_store.cc:269] 0426 13:20:30.159301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.930382 (+1771081us) service_pool.cc:225] Handling call
0426 13:20:31.957417 (+ 27035us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:31.959288 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:31.960351 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.896560 25024 rpcz_store.cc:276] 0426 13:20:30.754700 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.826232 (+ 71532us) service_pool.cc:225] Handling call
0426 13:20:30.826620 (+   388us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:30.828443 (+  1823us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:30.828482 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:30.829025 (+   543us) log.cc:842] Serialized 26 byte log entry
0426 13:20:31.041582 (+212557us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:31.041634 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.041718 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:31.715194 (+673476us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:31.786655 (+ 71461us) raft_consensus.cc:1671] finished
0426 13:20:31.786676 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:31.786956 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1055,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":4641664,"thread_start_us":212188,"threads_started":1,"wal-append.queue_time_us":397013}
I20240426 13:20:31.839867 28626 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:31.901124 24919 rpcz_store.cc:276] 0426 13:20:30.159620 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.806483 (+1646863us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:31.906461 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.910212 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:31.968187 24952 raft_consensus.cc:2362] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11. Re-sending same reply.
W20240426 13:20:31.911341 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93912.4us
I20240426 13:20:31.896807 28686 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:31.901131 28846 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:31.899188 28794 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:31.912886 24953 rpcz_store.cc:276] 0426 13:20:28.231029 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.231287 (+   258us) service_pool.cc:225] Handling call
0426 13:20:28.260116 (+ 28829us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.742304 (+1482188us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:31.826356 (+2084052us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:31.826609 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2934596480}
I20240426 13:20:31.826989 28681 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:31.915021 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:31.995250 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:31.995755 25620 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:31.923476 28777 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:31.997015 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.920467 28828 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:31.998432 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:31.920648 28798 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:31.922322 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.914705 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:31.933913 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:32.001713 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:32.002065 25105 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:31.935530 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:32.002707 25101 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:31.937026 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50066.8us
I20240426 13:20:32.003118 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:31.975512 28049 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:31.889139 28809 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240426 13:20:32.014051 28049 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:31.940253 28717 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.014636 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:32.016145 28844 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:31.943379 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.018115 28844 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:31.968709 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 672) took 1767 ms (client timeout 2066 ms). Trace:
I20240426 13:20:31.966027 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2246)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2249)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2248)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2250)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
W20240426 13:20:32.020169 24952 rpcz_store.cc:269] 0426 13:20:30.201152 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.958860 (+1757708us) service_pool.cc:225] Handling call
0426 13:20:31.968644 (+  9784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:31.974622 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:31.975844 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.976557 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:31.977484 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:31.841693 28728 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:31.915465 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:31.995955 24955 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:31.964839 28735 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:31.886539 28855 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.003701 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:31.937930 28746 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.051733 25028 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:32.009748 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2241) took 1231 ms (client timeout 1565 ms). Trace:
W20240426 13:20:32.080765 24953 rpcz_store.cc:269] 0426 13:20:30.778198 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.979543 (+1201345us) service_pool.cc:225] Handling call
0426 13:20:32.009650 (+ 30107us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:32.012585 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52907.4us
I20240426 13:20:32.082013 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:32.083318 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:31.943868 28731 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:32.016287 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:20:32.052242 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2235) took 1849 ms (client timeout 2033 ms). Trace:
I20240426 13:20:32.052395 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:20:32.086189 24955 rpcz_store.cc:269] 0426 13:20:30.203090 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.935064 (+1731974us) service_pool.cc:225] Handling call
0426 13:20:32.052186 (+117122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.086753 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:32.087349 25024 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:20:32.087263 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:32.014410 28809 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
W20240426 13:20:32.053371 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49749.9us
I20240426 13:20:32.018536 28756 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:32.055071 28631 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:32.008554 28828 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.009284 28681 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20240426 13:20:32.052963 24949 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:32.102797 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2237) took 1873 ms (client timeout 1933 ms). Trace:
I20240426 13:20:32.103111 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:32.103156 24949 rpcz_store.cc:269] 0426 13:20:30.229558 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.926709 (+1697151us) service_pool.cc:225] Handling call
0426 13:20:32.102735 (+176026us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.060202 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:32.061318 28842 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.826s	user 0.000s	sys 0.001s
I20240426 13:20:32.062568 28728 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.063577 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.102172 28631 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.185050 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.188622 28631 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:32.188999 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1846) took 2307 ms (client timeout 2763 ms). Trace:
W20240426 13:20:32.189414 24955 rpcz_store.cc:269] 0426 13:20:29.881314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.086884 (+2205570us) service_pool.cc:225] Handling call
0426 13:20:32.188928 (+102044us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.190410 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:32.185362 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2215) took 2658 ms (client timeout 1750 ms). Trace:
W20240426 13:20:32.191601 24951 rpcz_store.cc:269] 0426 13:20:29.527333 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.046039 (+518706us) service_pool.cc:225] Handling call
0426 13:20:32.103047 (+2057008us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b5c0028128c. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda93c5fdc1 00007fda93df9849 00007fda95259736 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:32.185292 (+ 82245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10377088}
I20240426 13:20:32.192763 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.193321 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.193545 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 673) took 1477 ms. Trace:
I20240426 13:20:32.193608 25618 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.193975 24951 rpcz_store.cc:276] 0426 13:20:30.715933 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.192532 (+1476599us) service_pool.cc:225] Handling call
0426 13:20:32.193492 (+   960us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.016261 28717 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.068020 28737 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:32.194111 24949 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:32.195696 28737 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:32.196354 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1842) took 2761 ms (client timeout 3229 ms). Trace:
W20240426 13:20:32.196727 24949 rpcz_store.cc:269] 0426 13:20:29.443533 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.104021 (+2660488us) service_pool.cc:225] Handling call
0426 13:20:32.196300 (+ 92279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.197336 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2240) took 1459 ms. Trace:
I20240426 13:20:32.067458 28852 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:32.069581 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1811) took 4063 ms (client timeout 2005 ms). Trace:
I20240426 13:20:32.074040 25620 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:32.074609 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:32.200136 25026 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.104341 24954 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:32.200268 24955 rpcz_store.cc:276] 0426 13:20:30.738147 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.190110 (+1451963us) service_pool.cc:225] Handling call
0426 13:20:32.197281 (+  7171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:31.999313 28777 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:31.978801 28745 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:32.089227 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.090775 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.009012 25624 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:32.093020 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:32.225809 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.156574 25024 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:32.226564 24955 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:32.227301 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 136622us
I20240426 13:20:32.227720 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:32.234313 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:32.194497 28846 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.197625 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.099027 25101 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:32.197763 28844 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:32.200971 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2238) took 1469 ms (client timeout 1636 ms). Trace:
I20240426 13:20:32.108856 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2005) took 2530 ms. Trace:
I20240426 13:20:32.201503 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.014503 28049 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:32.199836 24950 rpcz_store.cc:269] 0426 13:20:28.006532 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.006682 (+   150us) service_pool.cc:225] Handling call
0426 13:20:32.069382 (+4062700us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54000bf254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:32.069515 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37657856}
I20240426 13:20:32.068125 28815 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:32.065970 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:32.206574 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.253988 25105 raft_consensus.cc:595] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:31.997889 28794 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.228200 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.107045 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.239642 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2244) took 1096 ms. Trace:
I20240426 13:20:32.198179 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.241312 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.241701 25102 rpcz_store.cc:276] 0426 13:20:29.578954 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.579121 (+   167us) service_pool.cc:225] Handling call
0426 13:20:29.579488 (+   367us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.753733 (+174245us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:29.759293 (+  5560us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:29.759362 (+    69us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:29.759592 (+   230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.108561 (+2348969us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:32.108631 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:32.108790 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":395177728}
I20240426 13:20:32.259699 24949 rpcz_store.cc:276] 0426 13:20:31.143288 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.197366 (+1054078us) service_pool.cc:225] Handling call
0426 13:20:32.239565 (+ 42199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.260757 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:32.240368 24954 rpcz_store.cc:269] 0426 13:20:30.732063 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.048126 (+1316063us) service_pool.cc:225] Handling call
0426 13:20:32.200920 (+152794us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.261319 24952 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:32.262045 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:32.249296 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47867.7us
I20240426 13:20:32.254396 25105 raft_consensus.cc:603] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:32.253991 25023 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:32.098083 28746 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.104058 28660 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:32.090237 28809 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:32.097519 28756 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.097287 28706 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:32.258353 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.278796 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 676) took 1459 ms. Trace:
I20240426 13:20:32.279178 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.279914 24951 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:32.273213 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.255321 28697 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:32.264748 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.260190 28728 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:32.310246 25023 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:32.311627 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:32.103678 28540 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:32.263411 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2017) took 1494 ms. Trace:
I20240426 13:20:32.241326 28717 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:32.046679 28062 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.275426 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.279147 24955 rpcz_store.cc:276] 0426 13:20:30.819119 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.227813 (+1408694us) service_pool.cc:225] Handling call
0426 13:20:32.278660 (+ 50847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.242959 28745 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.261055 24953 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:32.255882 28824 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:32.290709 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:32.332947 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.253654 28815 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:32.267212 28735 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:32.313187 25026 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:32.101518 28817 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.167s
I20240426 13:20:32.327610 28713 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:32.324275 25101 rpcz_store.cc:276] 0426 13:20:30.768958 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.262646 (+1493688us) service_pool.cc:225] Handling call
0426 13:20:32.262939 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.263111 (+   172us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:32.263136 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:32.263179 (+    43us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:32.263215 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.263262 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:32.263362 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.278232 28706 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:32.270449 28733 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:32.209092 28777 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:32.351744 24949 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:32.279768 28756 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:32.097582 28626 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:32.355804 25028 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:32.404775 25028 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:32.359180 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2242) took 1580 ms (client timeout 1818 ms). Trace:
I20240426 13:20:32.313774 25024 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:32.324774 28840 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:20:32.364048 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:32.324036 24951 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:32.320173 28707 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:32.329155 28717 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.314023 28636 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:32.382364 28855 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.361202 28793 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:32.386055 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:32.387094 28870 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.388353 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.350258 28728 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.427474 24955 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:32.389546 28049 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.391777 28735 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:32.382201 24952 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:32.399454 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.399187 28852 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:32.405941 24953 rpcz_store.cc:269] 0426 13:20:30.778497 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.085517 (+1307020us) service_pool.cc:225] Handling call
0426 13:20:32.359110 (+273593us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.408699 25029 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 13:20:32.430382 25618 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:32.420766 28717 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:32.416266 28707 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:32.413347 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49407.1us
I20240426 13:20:32.417186 25023 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:32.410610 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.317606 28697 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.392670 28706 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:32.381404 28824 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.422811 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.406339 25101 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:32.428565 25624 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:32.429303 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2243) took 1650 ms (client timeout 1733 ms). Trace:
W20240426 13:20:32.427976 24920 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 6 similar messages]
I20240426 13:20:32.460975 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2019) took 1691 ms. Trace:
I20240426 13:20:32.405763 28713 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.411284 28737 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:32.445919 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:32.494808 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49036.1us
I20240426 13:20:32.430820 28735 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.419584 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 505) took 1115 ms. Trace:
I20240426 13:20:32.449636 25026 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:32.421061 28429 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:32.449856 24951 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:32.461472 25101 rpcz_store.cc:276] 0426 13:20:30.769589 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.395744 (+1626155us) service_pool.cc:225] Handling call
0426 13:20:32.401302 (+  5558us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.460723 (+ 59421us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.460923 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:32.460848 24952 rpcz_store.cc:269] 0426 13:20:30.778778 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.051491 (+1272713us) service_pool.cc:225] Handling call
0426 13:20:32.429232 (+377741us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.474874 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.460024 24949 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:32.448556 28876 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:20:32.481551 24920 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2254)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2255)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2251)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2264)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2266)
W20240426 13:20:32.451232 28877 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20240426 13:20:32.493512 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.496766 25031 rpcz_store.cc:276] 0426 13:20:31.304129 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.304462 (+   333us) service_pool.cc:225] Handling call
0426 13:20:31.304840 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:31.305101 (+   261us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:31.412216 (+107115us) log.cc:842] Serialized 14 byte log entry
0426 13:20:31.956876 (+544660us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b180049b340. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:20:32.419235 (+462359us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:32.419292 (+    57us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:32.419350 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.419412 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:32.419531 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":106947,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":4724,"raft.queue_time_us":54328,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8388,"spinlock_wait_cycles":22832640,"thread_start_us":461918,"threads_started":2,"wal-append.queue_time_us":866370}
I20240426 13:20:32.498922 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.574715 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.448371 28751 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:32.504067 24953 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:32.450973 28707 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:32.378751 28809 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.506335 25101 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:32.589570 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2020) took 1772 ms. Trace:
I20240426 13:20:32.590049 25101 rpcz_store.cc:276] 0426 13:20:30.817226 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.498353 (+1681127us) service_pool.cc:225] Handling call
0426 13:20:32.506085 (+  7732us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.589262 (+ 83177us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.589506 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.507815 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.591267 24951 raft_consensus.cc:2362] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11. Re-sending same reply.
I20240426 13:20:32.509018 28049 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:32.509454 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2014) took 2023 ms (client timeout 1913 ms). Trace:
I20240426 13:20:32.509567 28828 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:32.510358 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.601385 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.602130 24949 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:32.602344 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:32.512774 28882 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20240426 13:20:32.498903 28681 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20240426 13:20:32.564532 28429 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.615986 28429 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:32.514438 28885 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:20:32.499148 28735 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:32.622380 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.622627 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.623454 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:32.524809 28883 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:20:32.525614 28713 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.624959 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.528241 28697 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:32.528970 28728 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.544741 28852 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:32.547307 28691 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:32.547458 28817 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:32.547950 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 451768us
I20240426 13:20:32.505430 25621 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:32.568854 25023 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 13:20:32.562919 25624 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:32.571697 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:32.593344 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:32.597373 25096 rpcz_store.cc:269] 0426 13:20:30.486065 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.486226 (+   161us) service_pool.cc:225] Handling call
0426 13:20:32.509258 (+2023032us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:32.509389 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2612032384}
I20240426 13:20:32.425151 28815 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.660905 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:32.559056 28412 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:32.599603 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:32.604679 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.623993 25026 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.636087 24953 raft_consensus.cc:2362] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10. Re-sending same reply.
I20240426 13:20:32.638200 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:20:32.638828 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.643887 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.609637 28794 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:32.678526 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 387913us
I20240426 13:20:32.679484 24921 leader_election.cc:398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:32.679762 24921 leader_election.cc:401] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240426 13:20:32.675547 25624 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:32.662300 24951 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:32.683251 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.684093 24953 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:32.685962 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 335728us
I20240426 13:20:32.598954 28751 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:32.755820 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75541us
I20240426 13:20:32.645757 28691 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:32.757894 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:32.758088 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75860.5us
W20240426 13:20:32.735700 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50424.2us
I20240426 13:20:32.773562 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.775980 25624 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.775560 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:32.648602 28745 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.675144 28870 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.665802 28817 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.645851 28852 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.858952 28852 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:32.861153 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.861968 25022 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:32.862401 28751 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.863368 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:32.864599 28751 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:32.864656 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:32.865260 28751 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.865573 25617 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:32.865769 28751 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.869441 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:32.870527 28729 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:32.870993 28729 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:32.871439 28729 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:32.680856 28779 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:32.643122 28671 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
W20240426 13:20:32.770339 28746 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.500s	user 0.009s	sys 0.001s
I20240426 13:20:32.793109 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:32.793337 28360 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:32.793395 28660 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:32.798024 28691 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.800181 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:32.801533 28626 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20240426 13:20:32.808633 28878 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20240426 13:20:32.811344 28753 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:32.905994 28360 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:32.819636 24953 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:32.892719 28817 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:32.831354 28686 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.834s	user 0.002s	sys 0.006s
W20240426 13:20:32.832765 28884 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240426 13:20:32.835213 28798 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.816s	user 0.011s	sys 0.000s
I20240426 13:20:32.788278 28828 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20240426 13:20:32.845191 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2337) took 2811 ms. Trace:
I20240426 13:20:32.933226 25027 rpcz_store.cc:276] 0426 13:20:30.033843 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.034011 (+   168us) service_pool.cc:225] Handling call
0426 13:20:30.034381 (+   370us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:30.067116 (+ 32735us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:30.067147 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:30.067662 (+   515us) log.cc:842] Serialized 52 byte log entry
0426 13:20:31.912382 (+1844720us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b1800209fe0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:32.097923 (+185541us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:32.097963 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:32.098037 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:32.844789 (+746752us) raft_consensus.cc:1671] finished
0426 13:20:32.844819 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:32.845116 (+   297us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12950,"raft.queue_time_us":21842,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":29754,"spinlock_wait_cycles":18665344,"thread_start_us":185339,"threads_started":1,"wal-append.queue_time_us":2145333}
I20240426 13:20:32.755211 28794 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:32.700721 28815 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:32.898447 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:32.791484 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:32.784415 28631 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.595s	user 0.004s	sys 0.008s
I20240426 13:20:32.903570 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:32.943744 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:32.843305 28777 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.912925 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.950354 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.951179 24953 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:32.825753 28737 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
W20240426 13:20:32.799813 24950 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.000s
I20240426 13:20:32.958626 24950 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
W20240426 13:20:32.959431 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2245) took 1800 ms (client timeout 2147 ms). Trace:
W20240426 13:20:32.959911 24950 rpcz_store.cc:269] 0426 13:20:31.158803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.251014 (+1092211us) service_pool.cc:225] Handling call
0426 13:20:32.959363 (+708349us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:32.647058 28824 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:32.930069 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 129987us
I20240426 13:20:32.975342 25624 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:32.977830 25624 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 9. Ops: [7.9-7.9]
W20240426 13:20:32.981040 24995 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.893s (SENT)
I20240426 13:20:32.983101 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:32.981657 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:32.905160 28779 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:32.912086 28360 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:32.936380 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:32.912053 28753 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.757575 28412 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:32.999029 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:32.928651 28888 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:32.952878 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:32.967936 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:32.924289 24951 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:33.041101 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.043376 24953 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:32.907394 28735 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:32.991814 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:32.991926 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:32.926466 28717 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:32.947752 28758 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.059000 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.060048 24953 raft_consensus.cc:2362] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10. Re-sending same reply.
I20240426 13:20:33.061748 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.063503 24953 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:33.065122 24953 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:33.036011 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.112206 24952 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:33.112596 25615 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:32.936319 28686 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.041862 24951 raft_consensus.cc:2362] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7. Re-sending same reply.
I20240426 13:20:33.055101 25617 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:33.056790 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.115597 25624 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:32.944674 28794 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.072599 28360 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.061478 28844 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:33.020148 28798 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:33.155870 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:32.948655 28631 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:33.030406 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49405.3us
I20240426 13:20:33.106025 24950 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:33.158852 24995 connection.cc:232] RPC call timeout handler was delayed by -2.4087114860000156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2939s for a response
W20240426 13:20:33.159518 24995 connection.cc:232] RPC call timeout handler was delayed by -2.1477114859999347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3229s for a response
I20240426 13:20:33.159397 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:33.160238 24995 connection.cc:232] RPC call timeout handler was delayed by -1.6011536939998905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20240426 13:20:33.161579 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.161796 24950 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:33.162629 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1836) took 4055 ms. Trace:
I20240426 13:20:33.163050 24950 rpcz_store.cc:276] 0426 13:20:29.107146 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.160976 (+4053830us) service_pool.cc:225] Handling call
0426 13:20:33.161353 (+   377us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.162339 (+   986us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.162567 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.111310 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:33.113637 24955 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:33.114040 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1829) took 4804 ms. Trace:
I20240426 13:20:33.130466 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.131320 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:33.178546 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 415152us
I20240426 13:20:33.131982 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.179782 24951 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:33.183549 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:33.133190 25105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.000s	sys 0.013s
I20240426 13:20:33.166740 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.167207 28660 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:33.167989 28746 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:33.169476 28736 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:33.066982 28888 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:32.999799 28779 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:33.184551 25105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.000s	sys 0.013s
I20240426 13:20:33.192803 24954 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:33.177868 24953 rpcz_store.cc:276] 0426 13:20:28.309595 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.064554 (+4754959us) service_pool.cc:225] Handling call
0426 13:20:33.064907 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.113731 (+ 48824us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.113978 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.190851 24949 raft_consensus.cc:2354] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:33.191591 24951 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:33.190086 24950 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:33.070642 28758 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.071741 28751 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.207252 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2021) took 2331 ms. Trace:
I20240426 13:20:33.194075 28360 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:33.179693 28844 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:33.213889 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2212) took 3719 ms. Trace:
W20240426 13:20:33.219908 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.009s	sys 0.029s
W20240426 13:20:33.229128 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.009s	sys 0.029s
I20240426 13:20:33.228832 24951 rpcz_store.cc:276] 0426 13:20:29.494508 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.180509 (+3686001us) service_pool.cc:225] Handling call
0426 13:20:33.191335 (+ 10826us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:33.213516 (+ 22181us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.213819 (+   303us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:33.230268 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 550878us
I20240426 13:20:33.212702 28745 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:33.225742 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.235807 24953 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:33.232609 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2332) took 3823 ms. Trace:
I20240426 13:20:33.233934 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.185937 28753 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.205813 28412 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.246654 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.237046 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.271093 24951 raft_consensus.cc:2354] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:20:33.240465 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:33.243620 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.031s	sys 0.017s
W20240426 13:20:33.271978 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.037s	sys 0.021s
I20240426 13:20:33.271695 24949 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:33.285359 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.237506 25025 rpcz_store.cc:276] 0426 13:20:29.409308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.409446 (+   138us) service_pool.cc:225] Handling call
0426 13:20:29.438317 (+ 28871us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.788300 (+349983us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.232387 (+3444087us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:33.232555 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516886912}
I20240426 13:20:33.286733 24953 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:33.287791 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.288710 24953 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:33.289736 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:33.289865 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 306848us
W20240426 13:20:33.258680 28756 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.843s	user 0.006s	sys 0.006s
W20240426 13:20:33.237994 28429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.001s	sys 0.036s
I20240426 13:20:33.302345 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2387) took 1068 ms. Trace:
I20240426 13:20:33.214452 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1843) took 3744 ms. Trace:
W20240426 13:20:33.243952 28706 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.742s	user 0.008s	sys 0.003s
I20240426 13:20:33.310967 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:33.310925 24950 rpcz_store.cc:276] 0426 13:20:29.469895 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.163992 (+3694097us) service_pool.cc:225] Handling call
0426 13:20:33.189158 (+ 25166us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.213566 (+ 24408us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.214396 (+   830us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.319027 25026 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.268325 28717 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.310717 25031 rpcz_store.cc:276] 0426 13:20:32.233679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.588662 (+354983us) service_pool.cc:225] Handling call
0426 13:20:32.611912 (+ 23250us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:32.625578 (+ 13666us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:32.625621 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:32.626116 (+   495us) log.cc:842] Serialized 26 byte log entry
0426 13:20:33.272511 (+646395us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:33.272566 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.272658 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:33.302108 (+ 29450us) raft_consensus.cc:1671] finished
0426 13:20:33.302128 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:33.302289 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1071,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":3768,"thread_start_us":646043,"threads_started":1,"wal-append.queue_time_us":654892}
I20240426 13:20:33.314236 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.248396 28898 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.260632 28777 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:33.316332 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.326602 24949 raft_consensus.cc:2362] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10. Re-sending same reply.
I20240426 13:20:33.328614 25022 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20240426 13:20:33.331962 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 168993us
I20240426 13:20:33.239506 28900 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
W20240426 13:20:33.313632 28429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.003s	sys 0.045s
I20240426 13:20:33.332085 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:33.333065 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.317718 28686 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:33.308633 28735 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:33.337787 25069 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.521s (SENT)
I20240426 13:20:33.324559 28903 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.338680 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.346038 24951 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:33.336593 28891 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:33.320760 28631 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.340420 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.341393 25026 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:33.342197 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 108320us
I20240426 13:20:33.343284 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.360329 24949 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:33.362926 24951 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:33.363641 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.364060 24951 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.317133 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:33.365168 28902 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20240426 13:20:33.370976 24951 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:33.372200 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.333763 28794 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.309684 28713 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20240426 13:20:33.378059 24949 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:33.325012 28706 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.344790 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.273626 28844 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.384413 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:33.385643 25615 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:33.341820 28899 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:33.386197 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62774.3us
I20240426 13:20:33.386580 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.352182 28779 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.395062 24951 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:33.378916 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:33.373277 28886 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.251s	user 0.001s	sys 0.001s
I20240426 13:20:33.316871 28756 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.346264 28660 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:33.404125 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2210) took 3922 ms (client timeout 2042 ms). Trace:
W20240426 13:20:33.404577 24956 rpcz_store.cc:269] 0426 13:20:29.481689 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.403906 (+3922217us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:33.404058 (+   152us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:33.406061 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:33.390799 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 183085us
W20240426 13:20:33.391650 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54787us
I20240426 13:20:33.393692 25615 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:33.267140 28888 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.396586 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:33.414063 25026 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:33.388500 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.402477 28727 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.415525 28727 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:33.417101 28904 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20240426 13:20:33.418393 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2023) took 2440 ms. Trace:
W20240426 13:20:33.418843 28901 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20240426 13:20:33.418753 25096 rpcz_store.cc:276] 0426 13:20:30.977565 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.661803 (+1684238us) service_pool.cc:225] Handling call
0426 13:20:32.758808 (+ 97005us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:32.759240 (+   432us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:32.759283 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:32.790838 (+ 31555us) log.cc:842] Serialized 26 byte log entry
0426 13:20:33.233594 (+442756us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:33.233638 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.233709 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:33.418196 (+184487us) raft_consensus.cc:1671] finished
0426 13:20:33.418215 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:33.418341 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22014,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":24836,"thread_start_us":352718,"threads_started":1,"wal-append.queue_time_us":387908}
I20240426 13:20:33.420836 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:33.422080 25028 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:33.402890 25588 connection.cc:232] RPC call timeout handler was delayed by -1.3876240329999519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
I20240426 13:20:33.403753 25023 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:33.423499 25588 connection.cc:232] RPC call timeout handler was delayed by -1.3594666420000294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
I20240426 13:20:33.381369 28751 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:33.390820 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53073us
I20240426 13:20:33.422001 25098 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:33.394521 28891 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:33.392402 28758 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:33.410032 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.220816 25105 rpcz_store.cc:276] 0426 13:20:30.876087 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.618810 (+1742723us) service_pool.cc:225] Handling call
0426 13:20:32.619161 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.619424 (+   263us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:32.619734 (+   310us) log.cc:842] Serialized 14 byte log entry
0426 13:20:33.206773 (+587039us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:33.206843 (+    70us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:33.206924 (+    81us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.207022 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:33.207182 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":131,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":36,"raft.queue_time_us":5285,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4764,"thread_start_us":586637,"threads_started":1}
I20240426 13:20:33.404314 28660 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.330726 28898 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.392885 28828 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:33.420773 28779 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:33.408977 24995 leader_election.cc:398] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
W20240426 13:20:33.413048 28729 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.001s
I20240426 13:20:33.430912 24949 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:33.432281 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.448879 25026 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:33.432838 25025 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:33.420625 28707 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.819s	user 0.011s	sys 0.001s
I20240426 13:20:33.433127 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.434405 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:33.389397 28631 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20240426 13:20:33.428812 25097 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:33.456007 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:33.456542 25105 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:33.443946 24953 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:33.431479 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:33.352628 28903 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.448788 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.425571 28794 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.438628 25615 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.451052 24995 leader_election.cc:398] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:20:33.441488 28898 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.452214 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:33.452538 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.453096 28729 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.454136 28660 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.454807 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:33.454066 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:33.463914 25615 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:33.464160 24995 leader_election.cc:401] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:33.465262 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:33.467499 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:33.469482 25023 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:33.468442 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.463701 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.428267 28824 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20240426 13:20:33.486492 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.479725 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.482137 25023 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:33.490015 25025 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:33.458014 28900 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:33.425813 28908 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20240426 13:20:33.494774 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.477941 28828 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:33.465827 28691 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:33.483461 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.466600 28729 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.495806 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20240426 13:20:33.500583 24949 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:33.512260 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.512789 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1845) took 3957 ms. Trace:
I20240426 13:20:33.488629 28713 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:33.477578 28652 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.513214 24949 rpcz_store.cc:276] 0426 13:20:29.555973 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.485799 (+3929826us) service_pool.cc:225] Handling call
0426 13:20:33.500172 (+ 14373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.512559 (+ 12387us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.512737 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.509661 25615 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:33.511365 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.514858 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.513276 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:33.515633 25096 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:33.493276 28660 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:20:33.460307 28707 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.520865 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.512192 28680 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20240426 13:20:33.526219 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2022) took 2574 ms (client timeout 2110 ms). Trace:
W20240426 13:20:33.526629 25098 rpcz_store.cc:269] 0426 13:20:30.952141 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.993634 (+ 41493us) service_pool.cc:225] Handling call
0426 13:20:33.526039 (+2532405us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b54003e17d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:33.526147 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5532557184}
I20240426 13:20:33.532286 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.478593 28794 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.534967 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.535374 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.536221 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:33.527884 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.463389 28917 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20240426 13:20:33.514470 28891 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:33.545286 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 714) took 5789 ms. Trace:
I20240426 13:20:33.556228 25622 rpcz_store.cc:276] 0426 13:20:27.756340 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:27.756508 (+   168us) service_pool.cc:225] Handling call
0426 13:20:27.768757 (+ 12249us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.768193 (+1999436us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.544955 (+3776762us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54006a0ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:33.545201 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4163654016}
I20240426 13:20:33.550132 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.550359 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:33.552891 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66504.4us
I20240426 13:20:33.514076 28918 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:33.514714 28828 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:33.560918 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.519656 28706 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.559314 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.567214 28891 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:33.567493 28793 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.567682 28891 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:33.568289 28891 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:33.568053 28793 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.569633 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.562799 25096 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
W20240426 13:20:33.561692 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 154547us
I20240426 13:20:33.534168 28713 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:33.564584 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:33.564788 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2000) took 4032 ms. Trace:
I20240426 13:20:33.571362 25096 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:33.571786 25098 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:33.570165 28793 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.571951 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 654) took 2556 ms. Trace:
I20240426 13:20:33.573110 25096 rpcz_store.cc:276] 0426 13:20:31.015996 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.570937 (+2554941us) service_pool.cc:225] Handling call
0426 13:20:33.571212 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.571745 (+   533us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.571904 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.574106 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 655) took 2557 ms. Trace:
I20240426 13:20:33.574793 25096 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20240426 13:20:33.574440 25098 rpcz_store.cc:276] 0426 13:20:31.016671 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.571391 (+2554720us) service_pool.cc:225] Handling call
0426 13:20:33.571630 (+   239us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.573902 (+  2272us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.574061 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.575340 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 656) took 2558 ms. Trace:
I20240426 13:20:33.573279 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.576202 28639 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:33.576414 25098 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:33.571138 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.577405 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 658) took 1983 ms. Trace:
I20240426 13:20:33.578094 25098 rpcz_store.cc:276] 0426 13:20:31.594196 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.575999 (+1981803us) service_pool.cc:225] Handling call
0426 13:20:33.576269 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.577207 (+   938us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.577362 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.571120 25104 rpcz_store.cc:276] 0426 13:20:29.532156 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:29.532392 (+   236us) service_pool.cc:225] Handling call
0426 13:20:29.549146 (+ 16754us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:29.747717 (+198571us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.564579 (+3816862us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b54003e12d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:33.564730 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217582464}
I20240426 13:20:33.579614 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.580343 25098 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:33.578790 28639 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.581300 28793 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:33.580909 28639 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.581733 25098 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20240426 13:20:33.582340 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1667) took 1697 ms. Trace:
I20240426 13:20:33.582608 25098 rpcz_store.cc:276] 0426 13:20:31.884803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.581121 (+1696318us) service_pool.cc:225] Handling call
0426 13:20:33.581408 (+   287us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:33.582130 (+   722us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.582296 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.575676 25096 rpcz_store.cc:276] 0426 13:20:31.017371 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.574105 (+2556734us) service_pool.cc:225] Handling call
0426 13:20:33.574547 (+   442us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:33.575133 (+   586us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.575294 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.583501 28639 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.584479 25096 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:33.585276 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1668) took 1696 ms. Trace:
I20240426 13:20:33.585903 25096 rpcz_store.cc:276] 0426 13:20:31.888508 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.584093 (+1695585us) service_pool.cc:225] Handling call
0426 13:20:33.584323 (+   230us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.585076 (+   753us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.585232 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.585681 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.585238 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.587143 28660 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:33.587347 25096 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:33.587555 25098 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:33.587996 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1670) took 1689 ms. Trace:
I20240426 13:20:33.587750 28660 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:33.588402 25096 rpcz_store.cc:276] 0426 13:20:31.898633 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.586813 (+1688180us) service_pool.cc:225] Handling call
0426 13:20:33.587070 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.587795 (+   725us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.587954 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:33.588601 28660 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:33.582499 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:33.590355 28793 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:33.590296 28360 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:33.591028 28793 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:33.591787 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.591925 28793 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:33.592568 28733 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:33.594671 28733 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:33.595620 28733 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:33.595961 28733 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:33.596076 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:33.596536 28660 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.596777 28733 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.598207 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:33.599777 28733 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:33.549325 28729 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:33.548800 28810 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:33.519130 28652 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.521229 28735 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20240426 13:20:33.562186 28680 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:33.600914 28631 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20240426 13:20:33.563588 28779 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:33.608781 28909 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
W20240426 13:20:33.630510 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 201195us
I20240426 13:20:33.654125 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.600661 28736 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:33.609786 28706 raft_consensus.cc:2741] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20240426 13:20:33.599752 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:33.698498 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98846.4us
I20240426 13:20:33.699808 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:33.562826 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:33.701892 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139129us
I20240426 13:20:33.702538 28668 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:33.618925 28680 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:33.656797 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55165.6us
I20240426 13:20:33.615813 28810 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:33.603646 28900 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.616354 28917 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20240426 13:20:33.712184 28810 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:33.708227 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.726101 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.727015 25023 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:33.727890 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.708859 28680 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:33.735110 28660 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.730571 28930 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20240426 13:20:33.735944 28660 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:33.732414 28631 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20240426 13:20:33.737726 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 699) took 1797 ms. Trace:
I20240426 13:20:33.738303 28631 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.738535 25031 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:20:33.738970 28631 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:33.739827 28928 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20240426 13:20:33.735181 28652 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.707178 28929 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.741218 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.741392 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:33.737977 28899 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:33.718508 28908 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:33.718230 28706 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:33.738129 25619 rpcz_store.cc:276] 0426 13:20:31.940421 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.940691 (+   270us) service_pool.cc:225] Handling call
0426 13:20:31.940977 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:31.947492 (+  6515us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:32.206357 (+258865us) log.cc:842] Serialized 14 byte log entry
0426 13:20:32.206597 (+   240us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:32.206667 (+    70us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:32.206742 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.737473 (+1530731us) spinlock_profiling.cc:243] Waited 6.28 ms on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:33.737545 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:33.737669 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2646,"prepare.queue_time_us":428167,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":2278,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":15951,"spinlock_wait_cycles":14434560,"thread_start_us":258458,"threads_started":1}
I20240426 13:20:33.602056 28918 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.746297 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.748265 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:33.741775 28921 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:33.749452 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:33.753585 25622 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:33.753795 27911 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.750257 28739 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:33.742439 28735 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.727113 28935 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:33.799266 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58237.7us
I20240426 13:20:33.800285 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.743034 28903 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.802454 25618 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.725795 28412 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:33.751149 28908 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:33.790628 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2013) took 3625 ms. Trace:
I20240426 13:20:33.818089 25099 rpcz_store.cc:276] 0426 13:20:30.165251 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.199664 (+ 34413us) service_pool.cc:225] Handling call
0426 13:20:30.200003 (+   339us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:31.533295 (+1333292us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.790316 (+2257021us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:33.790540 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3060451200}
I20240426 13:20:33.817987 28908 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.808180 25622 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:33.813267 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.789177 28706 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:33.716599 28828 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.823330 25622 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:33.802193 28935 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:33.779891 28924 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20240426 13:20:33.820225 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.819597 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.833155 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:33.833320 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.789233 28933 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20240426 13:20:33.823422 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:33.814540 27911 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.825556 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.827714 25618 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:33.823519 28794 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.829085 28706 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.830341 28733 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:33.833492 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.834887 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:33.809454 28736 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:33.797518 28735 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.835919 28945 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20240426 13:20:33.823813 28707 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.840206 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.842666 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:33.752481 28929 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.782279 28779 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:33.848470 25099 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:33.857053 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:33.857730 27911 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.949730 25104 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:33.858879 28736 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.852629 28630 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:33.899029 28810 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.847394 28798 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:33.844874 28936 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:33.823415 28412 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:33.831799 28935 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:33.904143 28926 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:33.901405 28815 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.953s	user 0.008s	sys 0.013s
I20240426 13:20:33.920181 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.976804 25615 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:33.849839 28713 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.070s
I20240426 13:20:33.874871 25619 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:33.858942 28933 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:33.947036 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106934us
I20240426 13:20:33.846707 28891 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:33.873111 28631 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:33.955327 25622 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:33.951105 28733 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:33.844501 28652 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:33.958910 28735 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:33.977582 25105 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:33.978255 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:33.986986 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 657) took 2457 ms (client timeout 1547 ms). Trace:
W20240426 13:20:33.987393 25100 rpcz_store.cc:269] 0426 13:20:31.529250 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.596597 (+ 67347us) service_pool.cc:225] Handling call
0426 13:20:33.986796 (+2390199us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:33.986919 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340444928}
I20240426 13:20:33.972085 28630 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:33.978339 28926 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.009841 28062 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:34.017236 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:20:33.992537 28412 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:34.021368 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 701) took 1637 ms (client timeout 1745 ms). Trace:
W20240426 13:20:34.034298 25621 rpcz_store.cc:269] 0426 13:20:32.384333 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.385746 (+  1413us) service_pool.cc:225] Handling call
0426 13:20:34.014452 (+1628706us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b5400380c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.021308 (+  6856us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186290048}
W20240426 13:20:34.021529 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2375) took 2215 ms (client timeout 2091 ms). Trace:
I20240426 13:20:33.986747 28707 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
W20240426 13:20:34.021862 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2214) took 4522 ms (client timeout 1895 ms). Trace:
I20240426 13:20:34.013969 28631 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:34.022293 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:34.027557 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 689) took 2739 ms (client timeout 1870 ms). Trace:
W20240426 13:20:34.044345 25616 rpcz_store.cc:269] 0426 13:20:31.287803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.287927 (+   124us) service_pool.cc:225] Handling call
0426 13:20:34.013251 (+2725324us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.027497 (+ 14246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1188113792}
I20240426 13:20:34.028281 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 506) took 5044 ms. Trace:
I20240426 13:20:34.028534 28652 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:34.027331 28633 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
W20240426 13:20:34.030212 25097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1666) took 2145 ms (client timeout 2158 ms). Trace:
I20240426 13:20:34.030727 28933 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:33.947671 28793 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.035866 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:33.954349 28680 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:34.035933 25030 rpcz_store.cc:269] 0426 13:20:31.805721 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.943138 (+137417us) service_pool.cc:225] Handling call
0426 13:20:34.016962 (+2073824us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54004169d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.021461 (+  4499us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2113756544}
W20240426 13:20:34.037009 24948 rpcz_store.cc:269] 0426 13:20:29.499865 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:30.162814 (+662949us) service_pool.cc:225] Handling call
0426 13:20:34.017881 (+3855067us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.021800 (+  3919us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3678427520}
I20240426 13:20:34.061929 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.062443 27911 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:34.062966 27911 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:34.063408 27911 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:34.062458 24948 raft_consensus.cc:2362] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11. Re-sending same reply.
I20240426 13:20:34.065629 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.066315 24948 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:34.045408 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:34.045228 25623 rpcz_store.cc:276] 0426 13:20:28.983499 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:28.984086 (+   587us) service_pool.cc:225] Handling call
0426 13:20:30.786653 (+1802567us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:30.880793 (+ 94140us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.012996 (+3132203us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54005c1050. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.028201 (+ 15205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3964543360}
I20240426 13:20:34.026791 28739 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.029961 28918 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.047025 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2213) took 4552 ms. Trace:
W20240426 13:20:34.053436 25097 rpcz_store.cc:269] 0426 13:20:31.884527 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:31.929609 (+ 45082us) service_pool.cc:225] Handling call
0426 13:20:34.030053 (+2100444us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.030160 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3058894208}
I20240426 13:20:34.034336 28062 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.056761 25621 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20240426 13:20:34.057775 28894 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.842s	user 0.001s	sys 0.002s
I20240426 13:20:34.058573 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:33.988044 28798 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20240426 13:20:34.053298 28633 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:34.090706 25100 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:34.074496 24951 rpcz_store.cc:276] 0426 13:20:29.494814 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.395720 (+3900906us) service_pool.cc:225] Handling call
0426 13:20:33.396065 (+   345us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:33.396434 (+   369us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:33.396463 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:33.420333 (+ 23870us) log.cc:842] Serialized 52 byte log entry
0426 13:20:33.554807 (+134474us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:33.709302 (+154495us) log.cc:842] Serialized 14 byte log entry
0426 13:20:34.046266 (+336964us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.046672 (+   406us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:34.046734 (+    62us) raft_consensus.cc:1671] finished
0426 13:20:34.046755 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.046933 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18958,"prepare.queue_time_us":250915,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":23335,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":19125,"thread_start_us":563037,"threads_started":3,"wal-append.queue_time_us":138720}
I20240426 13:20:34.079618 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:34.094934 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:34.084497 25615 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:34.096453 25615 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:34.084928 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:34.085286 24919 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41288 dropped due to backpressure. The service queue is full; it has 50 items.
I20240426 13:20:34.087589 28680 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.053071 28903 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:34.098663 24919 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41288 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2288)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1871)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2294)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2292)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1863)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1861)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1870)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2286)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2300)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2302)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2265)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2266)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2272)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 679)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2276)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1856)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1860)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 680)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2287)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2295)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2298)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1866)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1867)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1868)
I20240426 13:20:34.088169 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:34.053437 28733 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:34.097199 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.104900 24951 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:34.106271 25615 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.107177 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:34.107676 25615 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:34.100162 25097 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:34.107187 27911 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.110306 25099 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:34.111366 27911 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.112506 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:34.112457 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.115998 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:34.116855 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:34.116897 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.117996 24951 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.105518 28707 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:34.119635 28947 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20240426 13:20:34.119935 28707 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.122133 28707 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.122671 28707 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.123176 28810 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.123119 28707 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.078982 28739 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.062566 28933 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.105265 28733 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.125500 25622 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:34.125684 28733 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:34.126610 28933 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.073854 28735 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:34.077816 28062 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:34.129211 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:34.132355 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.133610 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.134788 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.136204 24948 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:34.140872 28715 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:34.137470 28707 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.141201 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:34.145414 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2236) took 3941 ms (client timeout 3284 ms). Trace:
I20240426 13:20:34.140825 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.147442 28935 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.148298 24948 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:34.150022 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:34.150784 28922 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.501s	user 0.000s	sys 0.001s
I20240426 13:20:34.147923 24956 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:34.148720 28941 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20240426 13:20:34.135746 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:34.145931 24954 rpcz_store.cc:269] 0426 13:20:30.203821 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.261763 (+2057942us) service_pool.cc:225] Handling call
0426 13:20:34.145134 (+1883371us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54000a7dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.145327 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2138511232}
I20240426 13:20:34.155344 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:34.157660 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75142.8us
I20240426 13:20:34.164074 28926 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.164352 28739 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.164566 28935 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.165035 28929 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.167333 24951 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:34.167785 25615 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:20:34.164785 28891 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.177057 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.179030 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.179104 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.165061 28925 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20240426 13:20:34.182734 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:34.657244 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 474389us
I20240426 13:20:34.279541 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 520) took 1203 ms. Trace:
I20240426 13:20:34.201515 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:34.200939 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:34.686529 25615 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
W20240426 13:20:34.687811 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 486840us
I20240426 13:20:34.688261 28049 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:34.689596 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2409) took 1283 ms. Trace:
I20240426 13:20:34.215407 25097 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:34.693627 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1676) took 1254 ms. Trace:
I20240426 13:20:34.694038 25097 rpcz_store.cc:276] 0426 13:20:33.439422 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.178918 (+739496us) service_pool.cc:225] Handling call
0426 13:20:34.215165 (+ 36247us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.693028 (+477863us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.693399 (+   371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:34.694506 28779 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:34.665769 28641 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:34.156800 28630 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.697304 28779 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.697546 28641 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.698256 28779 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.664749 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:34.696942 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.701480 28779 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.702828 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2401) took 1514 ms. Trace:
I20240426 13:20:34.666838 28642 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:34.666913 28706 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:34.704877 25026 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:34.709671 28706 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.710582 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2026) took 2749 ms. Trace:
I20240426 13:20:34.703312 25027 rpcz_store.cc:276] 0426 13:20:33.188361 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.188556 (+   195us) service_pool.cc:225] Handling call
0426 13:20:33.188899 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.207870 (+ 18971us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:33.207906 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:33.207964 (+    58us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:33.208018 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.702226 (+1494208us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b54004155d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.702313 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.702571 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1770,"raft.queue_time_us":2259,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":6165,"spinlock_wait_cycles":38667008}
I20240426 13:20:34.648788 28966 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.408902 28793 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.712445 28790 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:34.178712 28736 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.669783 28652 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.669965 28753 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:34.206887 28680 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:34.253240 28714 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.671655 28735 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:34.671703 28707 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:34.393468 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46260 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:20:34.251039 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 728) took 1654 ms. Trace:
I20240426 13:20:34.646468 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.238222 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2030) took 1385 ms. Trace:
I20240426 13:20:34.681100 28742 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:34.681751 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1858) took 1380 ms (client timeout 1606 ms). Trace:
I20240426 13:20:34.683224 24949 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:34.682693 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.692201 28049 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:34.243522 28709 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:34.705753 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1695) took 2817 ms. Trace:
I20240426 13:20:34.690079 25022 rpcz_store.cc:276] 0426 13:20:33.405614 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.405784 (+   170us) service_pool.cc:225] Handling call
0426 13:20:34.689089 (+1283305us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.689344 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63453056}
I20240426 13:20:34.717737 25097 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:34.487763 28898 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:34.719785 25105 rpcz_store.cc:276] 0426 13:20:32.852942 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.949800 (+1096858us) service_pool.cc:225] Handling call
0426 13:20:33.950131 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.983599 (+ 33468us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.237880 (+254281us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.238126 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62703488}
W20240426 13:20:34.718950 24953 rpcz_store.cc:269] 0426 13:20:33.300966 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.314817 (+ 13851us) service_pool.cc:225] Handling call
0426 13:20:34.681208 (+1366391us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b54001018d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.681543 (+   335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162347776}
W20240426 13:20:34.726198 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.07039e+06us
I20240426 13:20:34.725971 28735 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.704663 28642 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.727401 28049 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.729007 28889 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:34.725277 28753 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.729872 28742 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.731021 28709 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:34.732681 28652 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.642367 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:34.589114 28851 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
W20240426 13:20:34.207850 28701 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20240426 13:20:34.157898 28412 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:34.490878 28917 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:34.718580 25620 rpcz_store.cc:276] 0426 13:20:32.596489 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.596758 (+   269us) service_pool.cc:225] Handling call
0426 13:20:32.597118 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.856427 (+259309us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:32.856473 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:32.856529 (+    56us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:32.856585 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.250699 (+1394114us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b5400473954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.250769 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.250964 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":595731200}
I20240426 13:20:34.740722 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.740864 28966 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.728673 24950 raft_consensus.cc:2362] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11. Re-sending same reply.
W20240426 13:20:34.606644 28908 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.775s	user 0.010s	sys 0.000s
I20240426 13:20:34.723421 25029 rpcz_store.cc:276] 0426 13:20:31.888159 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.392150 (+503991us) service_pool.cc:225] Handling call
0426 13:20:32.406753 (+ 14603us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:32.432745 (+ 25992us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.705238 (+2272493us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b54000564d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.705494 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31919488}
W20240426 13:20:34.739789 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47768us
I20240426 13:20:34.754302 28709 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.756182 28680 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:34.756881 28889 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.757838 28680 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.758781 28756 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.250193 28681 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
W20240426 13:20:34.604964 28959 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.429s	user 0.001s	sys 0.000s
I20240426 13:20:34.744812 28753 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.759763 28889 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:34.764750 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2282) took 1448 ms (client timeout 1665 ms). Trace:
I20240426 13:20:34.472774 28727 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.249600 28745 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:34.406844 28659 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.775041 25622 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:34.775553 28837 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.775884 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:34.755910 28709 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:34.773943 24950 rpcz_store.cc:269] 0426 13:20:33.303669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.681956 (+1378287us) service_pool.cc:225] Handling call
0426 13:20:34.752197 (+ 70241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:34.778553 24954 raft_consensus.cc:2362] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11. Re-sending same reply.
I20240426 13:20:34.784099 28642 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.786527 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.786684 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:34.398506 28846 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.791712 28742 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.792323 28745 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:34.793066 28837 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:34.787380 28715 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.581s	user 0.004s	sys 0.008s
I20240426 13:20:34.787395 24953 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:34.771000 28706 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.782748 28641 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.804838 25615 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:34.806461 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2231) took 5041 ms (client timeout 1702 ms). Trace:
I20240426 13:20:34.806540 28837 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.711043 25098 rpcz_store.cc:276] 0426 13:20:31.960982 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.588728 (+1627746us) service_pool.cc:225] Handling call
0426 13:20:33.588946 (+   218us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.697803 (+108857us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:33.715870 (+ 18067us) log.cc:842] Serialized 14 byte log entry
0426 13:20:33.807252 (+ 91382us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:33.807319 (+    67us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:33.807386 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.058909 (+251523us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b18002b63c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:20:34.710067 (+651158us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.710141 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.710396 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":20149,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":498,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":4130,"spinlock_wait_cycles":253957632,"thread_start_us":90451,"threads_started":1,"wal-append.queue_time_us":115642}
I20240426 13:20:34.690166 25617 rpcz_store.cc:276] 0426 13:20:33.075903 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.132551 (+ 56648us) service_pool.cc:225] Handling call
0426 13:20:34.021769 (+889218us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:34.024700 (+  2931us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:34.024741 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:34.029471 (+  4730us) log.cc:842] Serialized 26 byte log entry
0426 13:20:34.170359 (+140888us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:34.170769 (+   410us) log.cc:842] Serialized 14 byte log entry
0426 13:20:34.171374 (+   605us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.171477 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:34.172144 (+   667us) raft_consensus.cc:1671] finished
0426 13:20:34.172168 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.279289 (+107121us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b5400380c50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.279461 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2510,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":4601,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":264,"spinlock_wait_cycles":2046048384,"thread_start_us":139638,"threads_started":1,"wal-append.queue_time_us":140925}
W20240426 13:20:34.813525 24947 rpcz_store.cc:269] 0426 13:20:29.764705 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.806087 (+5041382us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:34.806260 (+   173us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:34.812934 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:34.751616 28707 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.800096 28898 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:34.821843 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.823019 24947 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
W20240426 13:20:34.823659 28733 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.697s	user 0.008s	sys 0.008s
I20240426 13:20:34.824191 28717 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:34.824664 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:34.826139 28716 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.826843 28049 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.827414 28652 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.827240 28715 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:34.827898 28966 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:34.839249 28659 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:34.840108 28633 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.842659 28633 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:34.844264 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 70048.4us
I20240426 13:20:34.845593 28809 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.836580 28716 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.845501 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.305720 28062 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.831549 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:34.847921 25098 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:34.854456 28062 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.855630 25098 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:34.856756 28822 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:34.857930 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.606s (SENT) [suppressed 43 similar messages]
I20240426 13:20:34.856774 28049 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.837363 28917 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:34.859131 28049 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.860435 28917 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.861661 28049 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.857533 28822 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:34.862540 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 220617us
W20240426 13:20:34.862838 28890 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.647s	user 0.002s	sys 0.003s
I20240426 13:20:34.863698 28851 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:34.865293 25096 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:34.865736 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:34.867063 28903 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:34.867518 28903 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.868669 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.871449 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.872328 28908 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.873049 24947 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.874282 28927 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:34.874709 28927 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:34.875051 28908 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.883596 28736 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.884150 28681 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:34.886006 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:34.886297 28681 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:34.887789 28062 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:34.888293 28062 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.888725 28062 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:34.889026 28898 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.890936 28898 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.868891 24948 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:34.875845 28927 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.897341 28731 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.897991 28731 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.899619 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2293) took 1072 ms. Trace:
I20240426 13:20:34.900000 24954 rpcz_store.cc:276] 0426 13:20:33.827219 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.180424 (+353205us) service_pool.cc:225] Handling call
0426 13:20:34.899421 (+718997us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b54000bf254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.899551 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111692544}
I20240426 13:20:34.901399 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.887789 28681 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:34.902663 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.867882 28903 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:34.884812 28808 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:34.921849 28808 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:34.922286 28808 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:34.904965 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1844) took 5387 ms. Trace:
I20240426 13:20:34.924762 24956 rpcz_store.cc:276] 0426 13:20:29.517054 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.405318 (+3888264us) service_pool.cc:225] Handling call
0426 13:20:34.147720 (+742402us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.155950 (+  8230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.904575 (+748625us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54000a7dd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.904776 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1706617088}
I20240426 13:20:34.925654 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2018) took 4156 ms. Trace:
I20240426 13:20:34.926666 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.928251 24956 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:34.929309 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2286) took 1105 ms. Trace:
I20240426 13:20:34.929733 24956 rpcz_store.cc:276] 0426 13:20:33.824351 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.926247 (+1101896us) service_pool.cc:225] Handling call
0426 13:20:34.929250 (+  3003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:34.053507 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:34.849722 28733 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:34.931051 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.931999 24956 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:34.912779 24947 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:34.914547 28900 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:34.861663 28917 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:34.932579 28733 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.917773 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2028) took 2155 ms. Trace:
I20240426 13:20:34.898133 28927 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:34.884543 28822 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.952466 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:34.868762 28707 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:34.908880 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.44008e+06us
I20240426 13:20:34.900297 28731 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:34.934443 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.929538 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 522) took 1181 ms. Trace:
I20240426 13:20:34.935389 28900 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:34.935851 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2232) took 4772 ms (client timeout 3510 ms). Trace:
I20240426 13:20:34.937872 25026 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:20:34.938252 25022 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20240426 13:20:34.938712 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2251) took 2866 ms (client timeout 1986 ms). Trace:
I20240426 13:20:34.932778 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2300) took 1086 ms. Trace:
I20240426 13:20:34.940299 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2029) took 2177 ms. Trace:
I20240426 13:20:34.941226 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:34.941569 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:34.942662 25622 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 13:20:34.942977 25620 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:20:34.943342 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:34.943712 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:34.944626 25025 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:34.959246 24952 rpcz_store.cc:269] 0426 13:20:32.071985 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.575813 (+503828us) service_pool.cc:225] Handling call
0426 13:20:34.938321 (+2362508us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.938535 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1133533568}
I20240426 13:20:34.946981 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 521) took 1200 ms. Trace:
I20240426 13:20:34.917668 25098 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20240426 13:20:34.949012 24953 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:34.949688 25096 rpcz_store.cc:276] 0426 13:20:32.761950 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.589437 (+827487us) service_pool.cc:225] Handling call
0426 13:20:33.589670 (+   233us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.865737 (+1276067us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.917484 (+ 51747us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.917713 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2932417792}
I20240426 13:20:34.926195 25102 rpcz_store.cc:276] 0426 13:20:30.769255 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.263280 (+1494025us) service_pool.cc:225] Handling call
0426 13:20:33.986991 (+1723711us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.987237 (+   246us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:33.987266 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:33.987308 (+    42us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:33.987337 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:33.987406 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.925211 (+937805us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540012d4d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.925426 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3963442944}
I20240426 13:20:34.953841 25618 rpcz_store.cc:276] 0426 13:20:33.747878 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.747989 (+   111us) service_pool.cc:225] Handling call
0426 13:20:34.929211 (+1181222us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.929371 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24104704}
I20240426 13:20:34.957177 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2217) took 5373 ms. Trace:
I20240426 13:20:34.960323 24956 rpcz_store.cc:276] 0426 13:20:33.846682 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.930739 (+1084057us) service_pool.cc:225] Handling call
0426 13:20:34.932718 (+  1979us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:34.957201 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2033) took 1595 ms. Trace:
I20240426 13:20:34.959493 25099 rpcz_store.cc:276] 0426 13:20:32.762427 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.948736 (+1186309us) service_pool.cc:225] Handling call
0426 13:20:33.949114 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.110789 (+161675us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.939703 (+828914us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.940079 (+   376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":370221952}
W20240426 13:20:34.957867 24955 rpcz_store.cc:269] 0426 13:20:30.163424 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.431381 (+2267957us) service_pool.cc:225] Handling call
0426 13:20:34.935430 (+2504049us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.935629 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1241276288}
I20240426 13:20:34.996547 28423 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240426 13:20:34.997218 27911 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:34.997465 25615 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:34.983219 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2301) took 1109 ms. Trace:
I20240426 13:20:35.003521 24954 rpcz_store.cc:276] 0426 13:20:33.874006 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.934148 (+1060142us) service_pool.cc:225] Handling call
0426 13:20:34.983151 (+ 49003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.004473 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.005232 24954 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.005776 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.006047 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2219) took 5363 ms. Trace:
I20240426 13:20:35.006863 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:34.875607 28745 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.007994 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.021490 25102 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:35.021646 28745 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:34.988090 25100 rpcz_store.cc:276] 0426 13:20:33.361287 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.988526 (+627239us) service_pool.cc:225] Handling call
0426 13:20:34.021384 (+ 32858us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.107934 (+ 86550us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.956722 (+848788us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b540039ec54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:34.957027 (+   305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158860672}
I20240426 13:20:35.022779 28423 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.023732 28423 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:35.025091 28969 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.069s	user 0.004s	sys 0.000s
I20240426 13:20:35.026954 28423 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.013025 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.029047 25097 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:35.031313 28716 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.006534 24954 rpcz_store.cc:276] 0426 13:20:29.642998 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.004458 (+5361460us) service_pool.cc:225] Handling call
0426 13:20:35.004955 (+   497us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.005759 (+   804us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.005967 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.033712 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 660) took 3161 ms. Trace:
I20240426 13:20:35.034162 25104 rpcz_store.cc:276] 0426 13:20:31.872347 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.580079 (+1707732us) service_pool.cc:225] Handling call
0426 13:20:33.949555 (+369476us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.950085 (+   530us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.033364 (+1083279us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54003ef8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:35.033558 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":848835328}
I20240426 13:20:35.035027 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.036190 28659 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:35.036404 28660 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.036957 28660 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.039572 28707 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:35.040450 28707 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.041391 28900 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.042971 28707 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.043855 25617 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:35.016522 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.047109 24952 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.013999 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.048559 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.039687 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2024) took 4037 ms. Trace:
I20240426 13:20:35.018414 28940 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:35.014664 24956 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:35.018760 28846 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:35.050848 28940 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:35.017190 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:34.993788 28753 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.052850 24955 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.051957 28846 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:35.054374 28846 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.055712 28846 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:35.039566 28660 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.008884 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2382) took 2961 ms. Trace:
I20240426 13:20:35.057478 25028 rpcz_store.cc:276] 0426 13:20:32.047691 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.429802 (+382111us) service_pool.cc:225] Handling call
0426 13:20:32.448708 (+ 18906us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:33.422643 (+973935us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.008456 (+1585813us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:35.008713 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2238209792}
I20240426 13:20:35.051810 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2218) took 5443 ms. Trace:
I20240426 13:20:35.058820 24947 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.058933 24956 rpcz_store.cc:276] 0426 13:20:29.608190 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.014067 (+5405877us) service_pool.cc:225] Handling call
0426 13:20:35.014415 (+   348us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.051554 (+ 37139us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.051735 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:34.987792 24949 rpcz_store.cc:276] 0426 13:20:29.584047 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.515166 (+3931119us) service_pool.cc:225] Handling call
0426 13:20:34.682663 (+1167497us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.722052 (+ 39389us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.956968 (+234916us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001018d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:34.957129 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2615054720}
I20240426 13:20:35.062034 25027 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 13:20:35.022810 25102 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:35.059813 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.097699 24947 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:35.069702 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:35.069766 25069 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.665s (SENT)
I20240426 13:20:35.084064 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2292) took 1160 ms. Trace:
I20240426 13:20:35.099087 24953 rpcz_store.cc:276] 0426 13:20:33.827074 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.752687 (+925613us) service_pool.cc:225] Handling call
0426 13:20:34.987671 (+234984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.100046 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.091660 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.016844 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1209) took 4200 ms. Trace:
I20240426 13:20:35.102406 25096 rpcz_store.cc:276] 0426 13:20:30.816867 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.014903 (+4198036us) service_pool.cc:225] Handling call
0426 13:20:35.016746 (+  1843us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.062292 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.074887 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2016) took 4306 ms. Trace:
I20240426 13:20:35.068688 24951 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:35.101399 24948 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:35.061810 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.103157 24952 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:35.063880 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.115427 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:35.115671 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.052999 28940 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.116461 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.116580 24952 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:35.116999 28940 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:35.117174 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:35.103016 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.117631 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.120369 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:35.121044 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:35.125036 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.112800 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2303) took 1004 ms. Trace:
I20240426 13:20:35.060271 28900 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:35.125772 24948 rpcz_store.cc:276] 0426 13:20:34.108716 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.156790 (+ 48074us) service_pool.cc:225] Handling call
0426 13:20:35.106042 (+949252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.074208 28753 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.127210 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:35.130029 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 272136us
I20240426 13:20:35.126914 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.050587 25101 rpcz_store.cc:276] 0426 13:20:31.002686 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.709607 (+1706921us) service_pool.cc:225] Handling call
0426 13:20:32.760478 (+ 50871us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:32.781564 (+ 21086us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:32.781609 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:32.797115 (+ 15506us) log.cc:842] Serialized 26 byte log entry
0426 13:20:34.739494 (+1942379us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b18002b6b40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:34.846169 (+106675us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:34.846222 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.846348 (+   126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:35.031791 (+185443us) raft_consensus.cc:1671] finished
0426 13:20:35.031810 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.032024 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20594,"raft.queue_time_us":28191,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":188,"spinlock_wait_cycles":113018752,"thread_start_us":58879,"threads_started":1,"wal-append.queue_time_us":2230934}
I20240426 13:20:35.132611 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1681) took 1409 ms. Trace:
I20240426 13:20:34.986471 25624 rpcz_store.cc:276] 0426 13:20:33.746133 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:33.779903 (+ 33770us) service_pool.cc:225] Handling call
0426 13:20:33.780283 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:33.782627 (+  2344us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:33.930385 (+147758us) log.cc:842] Serialized 14 byte log entry
0426 13:20:34.060091 (+129706us) log.cc:842] Serialized 14 byte log entry
0426 13:20:34.060281 (+   190us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:34.060758 (+   477us) log.cc:842] Serialized 26 byte log entry
0426 13:20:34.079427 (+ 18669us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:34.079717 (+   290us) log.cc:842] Serialized 14 byte log entry
0426 13:20:34.103234 (+ 23517us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:34.103334 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:34.931372 (+828038us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b18004e6760. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:34.946627 (+ 15255us) raft_consensus.cc:1671] finished
0426 13:20:34.946657 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:34.946815 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36329,"prepare.queue_time_us":251582,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":3507,"raft.queue_time_us":46191,"raft.run_cpu_time_us":438,"raft.run_wall_time_us":28943,"spinlock_wait_cycles":25149824,"thread_start_us":223395,"threads_started":2,"wal-append.queue_time_us":143995}
I20240426 13:20:35.136277 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.136335 25028 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:35.071524 28731 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:20:35.138000 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.138558 28731 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:35.138958 28731 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:35.139976 28731 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.140475 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:35.141451 28731 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:34.983965 28903 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.144047 24954 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.144927 28903 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.145574 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.146401 25030 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:35.146740 24954 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:35.147212 28822 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:35.147341 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.147445 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2220) took 5492 ms. Trace:
I20240426 13:20:35.147984 24954 rpcz_store.cc:276] 0426 13:20:29.655414 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.146122 (+5490708us) service_pool.cc:225] Handling call
0426 13:20:35.146455 (+   333us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.147203 (+   748us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.147398 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.148458 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:35.149873 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:35.150398 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80703.3us
I20240426 13:20:35.150625 25621 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:35.151628 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:35.151890 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:35.152231 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.149865 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2221) took 5494 ms. Trace:
I20240426 13:20:35.073863 28062 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.146622 28707 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.154706 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:35.149914 24951 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 11. Ops: []
I20240426 13:20:35.152944 24954 rpcz_store.cc:276] 0426 13:20:29.655766 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.148804 (+5493038us) service_pool.cc:225] Handling call
0426 13:20:35.149271 (+   467us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.149556 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.149579 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:35.149630 (+    51us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:35.149674 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.149721 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.149821 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.153303 25030 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:35.154054 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.153226 25027 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:35.156009 24950 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.160883 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1872) took 1001 ms. Trace:
I20240426 13:20:35.162928 24954 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:35.162765 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.163185 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.163486 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2223) took 5507 ms. Trace:
I20240426 13:20:35.163415 28808 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.164163 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.165033 28746 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:35.165452 28746 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:35.164132 28822 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.165716 28808 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.165827 28746 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:35.167340 28822 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.168359 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2227) took 5511 ms. Trace:
I20240426 13:20:35.168856 24954 rpcz_store.cc:276] 0426 13:20:29.657358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.162366 (+5505008us) service_pool.cc:225] Handling call
0426 13:20:35.162711 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.167990 (+  5279us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.168289 (+   299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.171556 24953 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.171613 24947 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:35.165432 25030 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240426 13:20:35.172245 28707 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.163964 24951 rpcz_store.cc:276] 0426 13:20:29.656306 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.149445 (+5493139us) service_pool.cc:225] Handling call
0426 13:20:35.149762 (+   317us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.163259 (+ 13497us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.163421 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.173540 24947 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:35.174130 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 665) took 5449 ms. Trace:
I20240426 13:20:35.174614 28707 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:35.174898 24851 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 175) took 25816 ms (client timeout 15000 ms). Trace:
I20240426 13:20:35.174455 24947 rpcz_store.cc:276] 0426 13:20:29.724367 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.173022 (+5448655us) service_pool.cc:225] Handling call
0426 13:20:35.173329 (+   307us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.173912 (+   583us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.174080 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.175942 25031 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:35.176295 25025 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.176334 24951 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 10. Ops: []
I20240426 13:20:35.130020 28900 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:35.125430 28809 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:35.161314 24950 rpcz_store.cc:276] 0426 13:20:34.159288 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.069389 (+910101us) service_pool.cc:225] Handling call
0426 13:20:35.160819 (+ 91430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.150424 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:35.183213 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 666) took 5422 ms. Trace:
W20240426 13:20:35.184291 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43925.2us
I20240426 13:20:35.184084 24951 rpcz_store.cc:276] 0426 13:20:29.755623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.174261 (+5418638us) service_pool.cc:225] Handling call
0426 13:20:35.176175 (+  1914us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.176959 (+   784us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.177173 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.187389 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:35.104982 25103 rpcz_store.cc:276] 0426 13:20:30.768669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:32.051608 (+1282939us) service_pool.cc:225] Handling call
0426 13:20:32.061134 (+  9526us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:32.104270 (+ 43136us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:32.184405 (+ 80135us) log.cc:842] Serialized 14 byte log entry
0426 13:20:32.249989 (+ 65584us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:32.250064 (+    75us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:32.250135 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.074474 (+2824339us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:35.074530 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.074731 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":81146,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":4080,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":7867,"spinlock_wait_cycles":98702592,"thread_start_us":65153,"threads_started":1,"wal-append.queue_time_us":224649,"wal-append.run_cpu_time_us":2351,"wal-append.run_wall_time_us":1225343}
I20240426 13:20:35.188897 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:35.191643 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:35.188431 24955 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:35.191206 24949 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:35.195173 24953 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:35.184023 28900 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:35.133386 28660 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.199285 25022 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:35.199785 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:35.205262 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2228) took 5481 ms. Trace:
I20240426 13:20:35.206069 25027 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:35.206348 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.206831 28937 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:35.207155 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.207563 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.209089 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 664) took 5480 ms. Trace:
I20240426 13:20:35.205687 24954 rpcz_store.cc:276] 0426 13:20:29.721343 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.169986 (+5448643us) service_pool.cc:225] Handling call
0426 13:20:35.187970 (+ 17984us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.192009 (+  4039us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.202523 (+ 10514us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b540004b850. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:35.202684 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40500480}
I20240426 13:20:35.209435 24953 rpcz_store.cc:276] 0426 13:20:29.723948 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.172361 (+5448413us) service_pool.cc:225] Handling call
0426 13:20:35.195029 (+ 22668us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.199494 (+  4465us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.204206 (+  4712us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54002cb5d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:35.204340 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51401472}
I20240426 13:20:35.214308 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.221024 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.225899 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2265) took 2828 ms. Trace:
I20240426 13:20:35.226545 24956 rpcz_store.cc:276] 0426 13:20:32.397480 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.198587 (+2801107us) service_pool.cc:225] Handling call
0426 13:20:35.209360 (+ 10773us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.225452 (+ 16092us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.225494 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:35.225555 (+    61us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:35.225614 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.225690 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.225831 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.133183 25105 rpcz_store.cc:276] 0426 13:20:33.723001 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:34.754754 (+1031753us) service_pool.cc:225] Handling call
0426 13:20:34.755207 (+   453us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:34.755729 (+   522us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:34.807089 (+ 51360us) log.cc:842] Serialized 14 byte log entry
0426 13:20:35.132197 (+325108us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:35.132273 (+    76us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:35.132333 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.132414 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.132548 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20553,"prepare.queue_time_us":60017,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":1911,"raft.queue_time_us":50828,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":4642,"thread_start_us":294633,"threads_started":1,"wal-append.queue_time_us":382185}
I20240426 13:20:35.240655 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.241768 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.242105 24951 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:35.214577 28745 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:35.266175 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.220465 28680 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:35.278861 24948 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:35.284835 24956 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:35.287011 24952 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:35.286078 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1860) took 1953 ms. Trace:
I20240426 13:20:35.296205 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2266) took 2856 ms. Trace:
I20240426 13:20:35.296705 24952 rpcz_store.cc:276] 0426 13:20:32.431376 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.195799 (+2764423us) service_pool.cc:225] Handling call
0426 13:20:35.265555 (+ 69756us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.287547 (+ 21992us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.287769 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.295224 24955 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 13:20:35.288225 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2226) took 5624 ms. Trace:
I20240426 13:20:35.298306 24948 rpcz_store.cc:276] 0426 13:20:29.657096 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.182967 (+5525871us) service_pool.cc:225] Handling call
0426 13:20:35.245843 (+ 62876us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.281293 (+ 35450us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.281510 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.269174 24920 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41288 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 680)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2287)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2295)
Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2296)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2297)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2298)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1866)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1867)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1868)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2305)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2307)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 685)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 688)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1875)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1876)
I20240426 13:20:35.298708 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2272) took 2371 ms. Trace:
I20240426 13:20:35.292817 24956 rpcz_store.cc:276] 0426 13:20:33.332758 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.228659 (+1895901us) service_pool.cc:225] Handling call
0426 13:20:35.271924 (+ 43265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.285792 (+ 13868us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.286017 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.305156 24955 rpcz_store.cc:276] 0426 13:20:32.927778 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.201923 (+2274145us) service_pool.cc:225] Handling call
0426 13:20:35.267265 (+ 65342us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.298261 (+ 30996us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.298647 (+   386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.311089 24953 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.301751 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.250167 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 669) took 5493 ms. Trace:
I20240426 13:20:35.240953 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.267858 27911 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.268289 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2276) took 2148 ms. Trace:
I20240426 13:20:35.318498 24951 rpcz_store.cc:276] 0426 13:20:29.757231 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.190837 (+5433606us) service_pool.cc:225] Handling call
0426 13:20:35.221426 (+ 30589us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.242488 (+ 21062us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.249902 (+  7414us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:35.250100 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47101312}
I20240426 13:20:35.248847 28822 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:35.211776 28937 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:35.252014 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52164.4us
I20240426 13:20:35.251364 24950 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.252729 24949 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.276975 25027 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:35.249091 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.278456 28680 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:35.325695 25027 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:35.325912 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1856) took 2066 ms. Trace:
I20240426 13:20:35.324579 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.330178 24953 rpcz_store.cc:276] 0426 13:20:33.259504 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.249836 (+1990332us) service_pool.cc:225] Handling call
0426 13:20:35.310598 (+ 60762us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.325669 (+ 15071us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.325863 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.318866 24954 rpcz_store.cc:276] 0426 13:20:33.119630 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.210549 (+2090919us) service_pool.cc:225] Handling call
0426 13:20:35.249544 (+ 38995us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.267933 (+ 18389us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.267969 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:35.268011 (+    42us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:35.268058 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.268120 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.268241 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":839,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":1792}
W20240426 13:20:35.175395 24851 rpcz_store.cc:269] 0426 13:20:09.420591 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.420965 (+   374us) service_pool.cc:225] Handling call
0426 13:20:35.174651 (+25753686us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:26.609219 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:26.609984 (+   765us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:26.610053 (+    69us) write_op.cc:427] Acquired schema lock
0426 13:20:26.610077 (+    24us) tablet.cc:660] Decoding operations
0426 13:20:26.611540 (+  1463us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:26.611634 (+    94us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:26.611675 (+    41us) tablet.cc:683] Acquiring locks for 12 operations
0426 13:20:26.884759 (+273084us) tablet.cc:699] Row locks acquired
0426 13:20:26.891700 (+  6941us) write_op.cc:260] PREPARE: finished
0426 13:20:26.929022 (+ 37322us) write_op.cc:270] Start()
0426 13:20:26.950921 (+ 21899us) write_op.cc:275] Timestamp: P: 1714137626923040 usec, L: 0
0426 13:20:26.963811 (+ 12890us) op_driver.cc:331] REPLICATION: starting
0426 13:20:27.118210 (+154399us) log.cc:842] Serialized 4739 byte log entry
0426 13:20:27.791879 (+673669us) op_driver.cc:444] REPLICATION: finished
0426 13:20:28.234486 (+442607us) write_op.cc:300] APPLY: starting
0426 13:20:28.263665 (+ 29179us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:28.399413 (+135748us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:28.484870 (+ 85457us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:29.006318 (+521448us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:29.006807 (+   489us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0426 13:20:29.006877 (+    70us) write_op.cc:311] APPLY: finished
0426 13:20:29.007415 (+   538us) log.cc:842] Serialized 89 byte log entry
0426 13:20:29.529802 (+522387us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:29.547605 (+ 17803us) write_op.cc:446] Released schema lock
0426 13:20:29.582164 (+ 34559us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:29.652410 (+ 70246us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:29.653972 (+  1562us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":155775,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":228366,"apply.run_cpu_time_us":30902,"apply.run_wall_time_us":1547493,"num_ops":12,"prepare.queue_time_us":192747,"prepare.run_cpu_time_us":98342,"prepare.run_wall_time_us":669963,"raft.queue_time_us":216268,"raft.run_cpu_time_us":17766,"raft.run_wall_time_us":252142,"replication_time_us":812583,"thread_start_us":1002009,"threads_started":5,"wal-append.queue_time_us":784921,"wal-append.run_cpu_time_us":36295,"wal-append.run_wall_time_us":2519778}]]}
I20240426 13:20:35.345127 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 668) took 5573 ms. Trace:
I20240426 13:20:35.345332 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.346047 24956 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:35.346670 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 683) took 1938 ms. Trace:
I20240426 13:20:35.345527 24950 rpcz_store.cc:276] 0426 13:20:29.756920 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.187758 (+5430838us) service_pool.cc:225] Handling call
0426 13:20:35.251160 (+ 63402us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.329517 (+ 78357us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.329732 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.347086 24956 rpcz_store.cc:276] 0426 13:20:33.408011 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.317287 (+1909276us) service_pool.cc:225] Handling call
0426 13:20:35.345877 (+ 28590us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.346450 (+   573us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.346621 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.357226 25098 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.357880 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.359338 24953 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:35.344695 27911 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.364414 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2295) took 1536 ms. Trace:
I20240426 13:20:35.363453 24952 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:35.364773 24953 rpcz_store.cc:276] 0426 13:20:33.827508 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.330936 (+1503428us) service_pool.cc:225] Handling call
0426 13:20:35.359177 (+ 28241us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.364182 (+  5005us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.364367 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.365717 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 680) took 1958 ms. Trace:
I20240426 13:20:35.346699 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 679) took 2228 ms. Trace:
I20240426 13:20:35.366358 24949 rpcz_store.cc:276] 0426 13:20:33.108867 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.199128 (+2090261us) service_pool.cc:225] Handling call
0426 13:20:35.252527 (+ 53399us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.337329 (+ 84802us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.337552 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.366061 24952 rpcz_store.cc:276] 0426 13:20:33.407232 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.308829 (+1901597us) service_pool.cc:225] Handling call
0426 13:20:35.363302 (+ 54473us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.365521 (+  2219us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.365675 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:35.358124 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 143888us
I20240426 13:20:35.351506 24852 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 7, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.372586 24852 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.374171 24852 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.375898 25023 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:35.376188 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.376703 24950 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:35.376875 25097 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:35.377351 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2297) took 1549 ms. Trace:
I20240426 13:20:35.377681 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.377959 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.378571 25023 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:35.359784 28659 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:35.379812 28970 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20240426 13:20:35.324733 28822 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:35.380406 28822 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:35.380451 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43653.1us
I20240426 13:20:35.381434 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:35.382045 28906 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.385329 27911 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.385121 28900 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.386116 28745 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.386899 25027 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.387233 28745 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.336961 28680 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:35.389511 28937 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.392752 24952 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:35.393393 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1867) took 1348 ms. Trace:
I20240426 13:20:35.393736 24952 rpcz_store.cc:276] 0426 13:20:34.044806 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.367733 (+1322927us) service_pool.cc:225] Handling call
0426 13:20:35.392524 (+ 24791us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.393152 (+   628us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.393345 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.395020 24952 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20240426 13:20:35.395227 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.395522 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1868) took 1350 ms. Trace:
I20240426 13:20:35.395815 25100 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:35.396389 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.398025 24953 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:35.398371 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.398645 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1864) took 1354 ms. Trace:
I20240426 13:20:35.399125 25096 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:35.399078 24953 rpcz_store.cc:276] 0426 13:20:34.043813 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.397435 (+1353622us) service_pool.cc:225] Handling call
0426 13:20:35.397800 (+   365us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.398429 (+   629us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.398599 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.373023 24954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:20:35.400310 24954 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:20:35.400728 24954 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.405053 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:35.405687 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1866) took 1361 ms. Trace:
I20240426 13:20:35.405995 24949 rpcz_store.cc:276] 0426 13:20:34.044506 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.404357 (+1359851us) service_pool.cc:225] Handling call
0426 13:20:35.404780 (+   423us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.405447 (+   667us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.405631 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.407317 24949 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:35.379423 28659 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:35.400733 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 211071us
I20240426 13:20:35.409582 28659 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.377687 24950 rpcz_store.cc:276] 0426 13:20:33.827845 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.347779 (+1519934us) service_pool.cc:225] Handling call
0426 13:20:35.376515 (+ 28736us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.377134 (+   619us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.377300 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.411788 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.413323 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.414084 24950 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:35.400470 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.415082 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:35.415709 24950 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:35.416442 25070 connection.cc:232] RPC call timeout handler was delayed by -1.4995012260000067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20240426 13:20:35.421180 24951 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20240426 13:20:35.421576 24951 pending_rounds.cc:76] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:20:35.421842 24951 pending_rounds.cc:97] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20240426 13:20:35.422092 24951 raft_consensus.cc:2866] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:35.422806 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2290) took 1596 ms. Trace:
I20240426 13:20:35.423122 24948 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:35.423370 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.423964 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 681) took 2016 ms. Trace:
I20240426 13:20:35.424311 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.423126 24951 rpcz_store.cc:276] 0426 13:20:33.826762 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.327933 (+1501171us) service_pool.cc:225] Handling call
0426 13:20:35.420749 (+ 92816us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.422568 (+  1819us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.422749 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.425987 25105 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20240426 13:20:35.426003 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.426865 24951 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:35.426987 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.427731 25101 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:35.427769 28680 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.429962 28680 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.395917 24952 rpcz_store.cc:276] 0426 13:20:34.045067 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.394422 (+1349355us) service_pool.cc:225] Handling call
0426 13:20:35.394805 (+   383us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.395306 (+   501us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.395476 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.428246 24951 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:35.434301 24956 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240426 13:20:35.434981 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2298) took 1607 ms. Trace:
I20240426 13:20:35.435356 24956 rpcz_store.cc:276] 0426 13:20:33.827982 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.347815 (+1519833us) service_pool.cc:225] Handling call
0426 13:20:35.369800 (+ 21985us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.434720 (+ 64920us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.434923 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.424378 24948 rpcz_store.cc:276] 0426 13:20:33.407511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.313218 (+1905707us) service_pool.cc:225] Handling call
0426 13:20:35.370907 (+ 57689us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.423687 (+ 52780us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.423902 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:35.416466 25027 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:35.441008 28735 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20240426 13:20:35.442296 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61021.6us
I20240426 13:20:35.375640 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 4 to 9. New cstate: current_term: 9 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:35.445333 24852 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.447451 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:35.448060 25619 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:35.448556 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:35.451406 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.451542 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:35.451972 25022 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.452348 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:35.453356 24954 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.454346 28908 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:35.455334 28908 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.455790 28908 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.458196 28908 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.458806 28908 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:35.459453 24954 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.459286 28908 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:35.447010 24852 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 4 to 6, now has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:35.460070 25616 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:35.458527 24950 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
W20240426 13:20:35.460588 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 304756us
I20240426 13:20:35.460009 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2296) took 1632 ms. Trace:
I20240426 13:20:35.465370 24956 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20240426 13:20:35.465863 24956 pending_rounds.cc:76] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:20:35.466127 24956 pending_rounds.cc:97] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20240426 13:20:35.466430 24956 raft_consensus.cc:2866] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:35.231985 28660 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.471387 25100 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:35.470798 24949 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:35.463788 24954 rpcz_store.cc:276] 0426 13:20:33.827700 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.331055 (+1503355us) service_pool.cc:225] Handling call
0426 13:20:35.459914 (+128859us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":157}
I20240426 13:20:35.478780 24950 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:35.240516 28809 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.419822 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:35.487615 24948 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:35.489995 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:35.492332 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82168.7us
I20240426 13:20:35.492549 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.493666 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.494396 24952 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:35.496996 25031 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:35.499315 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.400512 28900 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.460573 28062 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.507550 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.516695 24852 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 10, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.518915 25025 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:35.500939 28903 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.516126 28940 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:35.547590 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2287) took 1721 ms. Trace:
I20240426 13:20:35.558393 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.532850 28898 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:35.564759 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:35.500571 28746 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.567612 25620 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20240426 13:20:35.568157 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 148396us
I20240426 13:20:35.575054 24852 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.581461 28777 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:35.581678 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.582119 28777 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:35.582660 28777 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:35.582768 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:35.586966 28660 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.587694 28660 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:35.589088 28943 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:35.593991 28898 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:35.595356 28809 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.596313 28062 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.597843 25619 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:35.602550 28855 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:35.627569 28870 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:35.589228 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.556847 24955 rpcz_store.cc:276] 0426 13:20:33.826198 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.436765 (+1610567us) service_pool.cc:225] Handling call
0426 13:20:35.511588 (+ 74823us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.547090 (+ 35502us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:35.547138 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:35.547198 (+    60us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:35.547276 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.547363 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:35.547522 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20044,"raft.queue_time_us":21141,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6239}
I20240426 13:20:35.616873 28906 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:35.540402 28707 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20240426 13:20:35.542675 28735 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20240426 13:20:35.561133 28900 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.543074 28908 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.624007 25027 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20240426 13:20:35.624322 28809 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.626761 25098 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:35.553783 28903 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:35.638187 25022 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:35.602401 24852 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:35.654501 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 149931us
I20240426 13:20:35.649773 25027 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 5
I20240426 13:20:35.658947 25027 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20240426 13:20:35.659406 25027 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:35.645344 28822 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.660627 28906 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:35.661809 28906 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:35.662137 28906 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.662401 28822 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.663074 28906 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.664021 28746 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:35.664458 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75346.9us
I20240426 13:20:35.665799 28903 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.665942 28735 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:35.668509 28906 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.668365 28809 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:35.669011 28777 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.669531 28908 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.670557 28809 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:20:35.672411 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100331us
I20240426 13:20:35.673208 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:35.619890 28680 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20240426 13:20:35.676121 28898 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:35.676395 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.677394 28898 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:35.677847 28898 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:35.678273 28898 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:35.678959 24852 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 5 to 10, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.681257 24852 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.682381 28917 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:35.669040 28900 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.686971 28900 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.688102 25624 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 12 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:35.628563 28062 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.688382 28809 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.688968 28809 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.689875 28062 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:35.690977 28977 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.105s	user 0.000s	sys 0.002s
I20240426 13:20:35.691249 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.682621 24852 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 5 to 6, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.656968 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:35.643966 28870 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.705524 25616 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:35.670902 28777 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.721431 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:35.709743 28062 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:35.690997 28809 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.719661 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 8 to 10, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 10 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:35.728025 25622 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:35.729794 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.717805 28846 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.732358 28735 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:35.721069 28900 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:35.750499 28716 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:35.753161 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.713478 28898 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.757229 28735 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:35.758679 28735 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:35.759626 28735 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.773787 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:35.785177 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.788656 24852 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.791990 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.753979 28777 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20240426 13:20:35.799446 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:35.800120 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.803078 25023 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:35.800701 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:35.803750 25025 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.747450 28846 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.775396 28943 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:35.795029 28927 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:35.813807 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:35.817378 25022 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:35.813580 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:35.820410 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.820637 25105 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:35.821575 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.822811 25101 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:35.823985 25099 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:35.824380 25099 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.827618 25616 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:20:35.815721 25622 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:35.770399 28900 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.835004 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:35.835394 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:35.837839 24955 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:35.802688 28903 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:35.836478 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80654.9us
I20240426 13:20:35.803016 28680 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:35.840314 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.841521 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:35.846516 24952 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:35.843817 25099 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.854334 27911 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:35.854840 27911 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.856977 24852 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.858618 25099 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.788363 28716 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
W20240426 13:20:35.858829 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93412.2us
I20240426 13:20:35.860332 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:35.864720 28846 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.865841 28846 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:35.866859 28716 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:35.867270 28716 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.868110 28716 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.869315 28716 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:35.869455 28900 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:35.870046 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.871351 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.872349 28758 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:35.874228 28758 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:35.874368 28777 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:35.875519 28777 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:35.861505 28906 raft_consensus.cc:2741] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20240426 13:20:35.875823 28777 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.877293 28777 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.878475 28777 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:35.872184 28659 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:35.880977 28659 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:35.881312 28659 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:35.882184 28659 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.883352 28659 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:35.874626 28758 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:35.873759 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:35.857832 28903 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.781760 28870 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.895449 25099 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:35.895481 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:35.900394 27911 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:35.872249 28429 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:20:35.906172 25029 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.020s	sys 0.006s
W20240426 13:20:35.906708 25029 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.020s	sys 0.006s
I20240426 13:20:35.885727 24852 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:35.907455 25103 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.010s	sys 0.018s
W20240426 13:20:35.909001 25103 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.010s	sys 0.018s
I20240426 13:20:35.888408 28758 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:35.913417 27911 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:35.915097 24852 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:35.897845 28822 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:35.903740 28906 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:35.919591 24955 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:35.933064 28809 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:35.934834 28626 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:35.935379 28900 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:35.936903 28917 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:35.937995 24852 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:35.957659 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45137.6us
I20240426 13:20:35.953403 28809 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:35.921600 28898 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:35.929538 28822 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:35.965931 28906 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:35.965265 24852 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:35.968413 28809 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:35.987032 28903 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:36.006099 24953 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.023s	sys 0.044s
I20240426 13:20:35.953635 28680 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
W20240426 13:20:36.009150 24953 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.023s	sys 0.045s
W20240426 13:20:36.020519 25589 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.016s	sys 0.050s
I20240426 13:20:36.022352 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 9 to 10, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:20:36.024561 25589 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.017s	sys 0.050s
W20240426 13:20:36.027056 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 586552us
I20240426 13:20:36.048310 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 7, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:36.052199 28899 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.004s	sys 0.025s
I20240426 13:20:36.055614 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:36.053159 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.009s	sys 0.020s
W20240426 13:20:36.073573 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.010s	sys 0.020s
I20240426 13:20:36.022173 28716 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20240426 13:20:36.076391 24852 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 8, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 8 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:36.086295 28927 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:36.086797 28927 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:36.094941 28855 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:20:36.059387 28680 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20240426 13:20:36.055845 28906 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.100889 28716 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:36.101616 28870 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:36.040928 28822 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.103804 28822 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:36.073328 28899 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.004s	sys 0.025s
I20240426 13:20:36.133195 25025 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:36.132825 24852 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 9 to 10, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:36.105700 24947 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
W20240426 13:20:36.135767 28937 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.006s	sys 0.020s
W20240426 13:20:36.144490 28937 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.006s	sys 0.020s
I20240426 13:20:36.145869 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 667) took 6389 ms. Trace:
I20240426 13:20:36.146224 24947 rpcz_store.cc:276] 0426 13:20:29.756273 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.183494 (+5427221us) service_pool.cc:225] Handling call
0426 13:20:35.201503 (+ 18009us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:36.144464 (+942961us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:36.145627 (+  1163us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:36.145796 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2079145344}
I20240426 13:20:36.147619 24852 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:36.149361 24852 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 8, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 8 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:36.151037 24852 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:36.152946 24852 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 5 to 9, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 9 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:36.154537 24852 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
W20240426 13:20:36.136461 28733 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.013s	sys 0.016s
W20240426 13:20:36.225903 28733 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.014s	sys 0.017s
W20240426 13:20:36.236236 28745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.008s	sys 0.028s
W20240426 13:20:36.238376 28745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.008s	sys 0.028s
I20240426 13:20:36.241441 28903 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.242015 28903 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.243999 28903 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.231204 28899 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.261345 28733 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:36.264171 28733 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:36.265775 28733 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.266796 28733 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:36.257182 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.000s	sys 0.055s
W20240426 13:20:36.268400 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.000s	sys 0.055s
W20240426 13:20:36.268771 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 713968us
W20240426 13:20:36.269503 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.009s	sys 0.047s
W20240426 13:20:36.269879 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.009s	sys 0.047s
W20240426 13:20:36.270285 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 709515us
I20240426 13:20:36.293483 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:36.239630 28927 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:36.295189 28062 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20240426 13:20:36.324299 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:36.330210 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:36.358696 25617 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.008s	sys 0.033s
W20240426 13:20:36.359678 25617 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.008s	sys 0.033s
I20240426 13:20:36.262755 28899 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:36.331688 28735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.002s	sys 0.032s
I20240426 13:20:36.326386 28626 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 13:20:36.379019 28735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.003s	sys 0.033s
W20240426 13:20:36.376996 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66572.9us
I20240426 13:20:36.371742 28937 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:36.327445 28062 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:36.327579 28940 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.018s	sys 0.048s
I20240426 13:20:36.383356 28626 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:36.375072 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.009s	sys 0.013s
W20240426 13:20:36.400861 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.010s	sys 0.013s
I20240426 13:20:36.377929 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:36.375110 28808 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:36.396831 28758 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.387603 28937 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:36.401268 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.002s	sys 0.035s
I20240426 13:20:36.409389 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1689) took 1093 ms. Trace:
W20240426 13:20:36.427440 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.002s	sys 0.036s
W20240426 13:20:36.383569 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.006s	sys 0.016s
W20240426 13:20:36.437573 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.006s	sys 0.016s
I20240426 13:20:36.437775 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1722) took 1075 ms. Trace:
W20240426 13:20:36.438066 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 617129us
W20240426 13:20:36.394379 28940 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.019s	sys 0.048s
W20240426 13:20:36.440831 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 523607us
W20240426 13:20:36.441335 25618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.007s	sys 0.011s
I20240426 13:20:36.427567 25103 rpcz_store.cc:276] 0426 13:20:35.316293 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.329578 (+ 13285us) service_pool.cc:225] Handling call
0426 13:20:35.358048 (+ 28470us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.387076 (+ 29028us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.387119 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:35.387598 (+   479us) log.cc:842] Serialized 26 byte log entry
0426 13:20:35.911661 (+524063us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:35.914619 (+  2958us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:35.914708 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:36.409197 (+494489us) raft_consensus.cc:1671] finished
0426 13:20:36.409219 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:36.409337 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17359,"raft.queue_time_us":18614,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":523685,"threads_started":1,"wal-append.queue_time_us":939073}
W20240426 13:20:36.446223 25618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.007s	sys 0.011s
I20240426 13:20:36.440037 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:36.416481 28735 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.402249 28626 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:36.413170 28899 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.423139 28937 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:36.458709 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:36.407028 28758 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:36.460741 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83162.4us
I20240426 13:20:36.450392 28808 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:36.477043 28808 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.480556 28900 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:36.480994 28900 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:36.481765 28900 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.483040 28900 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:36.438481 25030 rpcz_store.cc:276] 0426 13:20:35.362334 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.362465 (+   131us) service_pool.cc:225] Handling call
0426 13:20:35.372200 (+  9735us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.431584 (+ 59384us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.431618 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:35.478107 (+ 46489us) log.cc:842] Serialized 26 byte log entry
0426 13:20:36.074111 (+596004us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:36.074167 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:36.074304 (+   137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:36.423045 (+348741us) raft_consensus.cc:1671] finished
0426 13:20:36.423065 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:36.437676 (+ 14611us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72603,"raft.queue_time_us":59079,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"thread_start_us":545489,"threads_started":1,"wal-append.queue_time_us":903655}
W20240426 13:20:36.450294 28989 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.075s	user 0.000s	sys 0.005s
I20240426 13:20:36.447448 28940 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:36.471467 27911 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:36.486135 27911 consensus_queue.cc:1225] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:36.487217 28927 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.472151 28707 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.487836 28927 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:36.488142 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44670.8us
I20240426 13:20:36.460065 28898 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:36.485487 28940 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.488965 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.489749 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:36.490351 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.470618 28660 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:36.491236 28660 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:36.444448 28429 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:36.488016 28707 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.503068 28999 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:36.501117 28809 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:36.504179 27911 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:36.505175 28870 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:36.506242 28927 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.507751 28707 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:36.508854 28927 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.509533 28927 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.509954 28927 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.512245 28777 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:36.513998 28777 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:36.516253 28753 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:36.519600 28733 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:36.520416 28898 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:36.469416 28735 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.491636 28660 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:36.506520 27911 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 13:20:36.523463 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64570.5us
I20240426 13:20:36.527631 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.530594 25097 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:36.522717 28943 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.007s	sys 0.023s
W20240426 13:20:36.531749 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 471608us
I20240426 13:20:36.530670 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:36.515559 28777 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:36.541692 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:36.542431 25030 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:36.533332 28943 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.007s	sys 0.023s
I20240426 13:20:36.546947 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:36.554057 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:36.554648 25023 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:36.551445 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:36.488986 28899 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20240426 13:20:36.563056 25027 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:36.564543 24955 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:20:36.565515 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:36.538087 28731 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.014s	sys 0.017s
I20240426 13:20:36.532438 28707 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20240426 13:20:36.552726 28870 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:36.582967 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:36.584233 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.586095 25622 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20240426 13:20:36.610332 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63490us
I20240426 13:20:36.617707 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:36.615178 28870 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:36.616933 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:36.569582 28731 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.015s	sys 0.017s
I20240426 13:20:36.625824 24947 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:36.559988 28753 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.631528 25623 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:36.632809 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:36.633420 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:36.634899 25624 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:36.642937 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58821.9us
I20240426 13:20:36.645160 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.645884 24955 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:36.649518 24948 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:36.562646 28943 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:36.650854 25620 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:20:36.631652 28870 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.660358 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:36.663300 25615 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:36.653276 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 70388.3us
I20240426 13:20:36.653772 27911 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:36.639801 25023 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:36.666926 25621 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:36.668241 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:36.564004 28809 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20240426 13:20:36.645566 28999 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20240426 13:20:36.611048 28998 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:36.677544 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:36.678119 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2305) took 1956 ms. Trace:
I20240426 13:20:36.679622 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:36.681978 25616 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:36.683313 25620 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:36.684834 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:36.685528 24955 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:36.655174 29009 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:36.686730 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:36.674588 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 144010us
I20240426 13:20:36.690268 24948 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:36.676729 24947 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20240426 13:20:36.678617 24953 rpcz_store.cc:276] 0426 13:20:34.743424 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.399999 (+656575us) service_pool.cc:225] Handling call
0426 13:20:35.416829 (+ 16830us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:35.417121 (+   292us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:36.012294 (+595173us) log.cc:842] Serialized 14 byte log entry
0426 13:20:36.677778 (+665484us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:36.677841 (+    63us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:36.677902 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:36.677966 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:36.678067 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":138011,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":8211,"thread_start_us":1216992,"threads_started":3}
I20240426 13:20:36.705185 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:36.706379 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 235574us
I20240426 13:20:36.709275 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:36.712450 27911 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:36.626988 28626 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.716228 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:36.679702 28998 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:36.621372 28429 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:36.700176 29017 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.653986 28753 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.753628 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.759837 25097 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:36.762431 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:36.763162 24953 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:36.769379 29007 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20240426 13:20:36.769598 28997 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:36.770052 28997 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:36.770119 29011 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:36.770906 28997 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.772423 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:36.770066 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:36.783738 25615 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:36.788511 28735 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:36.801412 28626 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:36.802152 28626 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:36.802561 28626 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:36.802956 28626 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:36.803854 25025 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:36.660276 28898 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:36.744619 29004 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20240426 13:20:36.722574 28731 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:36.815285 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:36.816653 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:36.818492 25615 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20240426 13:20:36.723102 28998 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:36.740211 29016 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:36.829030 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:36.855844 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.856649 24956 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:36.858493 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.752362 28927 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.717278 28062 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.099s
I20240426 13:20:36.859961 24956 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
W20240426 13:20:36.861667 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45117.4us
I20240426 13:20:36.853224 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:36.867056 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:36.857924 24955 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:36.708317 29009 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.878898 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.886241 24954 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:36.886723 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:36.879343 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:36.887498 24956 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:36.887807 24951 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:36.878537 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:36.880391 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 204121us
I20240426 13:20:36.884049 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.889186 24947 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:36.880075 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.890245 24950 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:36.874963 24953 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:36.892830 29021 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.868431 29016 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.893584 29021 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.894614 28707 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:36.894825 29016 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:36.895074 28707 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:36.895483 28707 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:36.892879 29003 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20240426 13:20:36.889343 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:36.828441 28735 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:36.902135 25030 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:36.901506 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.903470 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.892227 24953 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20240426 13:20:36.907370 24953 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:36.910593 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:36.914842 25618 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 13:20:36.912765 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45820.7us
I20240426 13:20:36.914379 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:36.937841 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:36.913810 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 84885us
I20240426 13:20:36.880069 28855 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:36.945240 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:36.946549 28731 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:36.946805 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174460us
I20240426 13:20:36.949448 25097 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:36.950366 25619 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:36.910065 28903 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.902798 28997 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:36.960508 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:36.961566 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 148249us
I20240426 13:20:36.966796 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.904526 28927 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:36.965027 25024 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:36.937856 28716 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:36.968262 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:36.968189 28927 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.937012 28733 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:36.985857 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.807301 29017 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:36.989681 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.991420 25030 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:36.991915 25030 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:36.995558 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 46121.8us
I20240426 13:20:36.999451 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.001323 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:36.931279 28777 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20240426 13:20:36.870213 28943 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.018610 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:37.012240 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:37.021467 25098 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:37.016125 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:36.982280 29021 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.013516 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.937484 28899 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20240426 13:20:37.035966 27911 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.036180 25096 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:37.036852 25102 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240426 13:20:37.018494 25099 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:37.044445 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.007128 28735 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:37.031289 25030 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.042176 25101 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:37.073179 27911 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.074884 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.073221 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:37.092655 25030 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:37.036702 29014 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20240426 13:20:37.096895 27911 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:37.161150 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:36.997975 28733 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.152781 29018 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:37.032446 28659 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20240426 13:20:37.156447 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69793.6us
I20240426 13:20:37.159235 25098 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:37.015050 28900 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:37.165618 25101 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:37.161226 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.210299 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:36.983503 28903 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.162113 25099 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:37.074085 28735 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.170841 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033386
I20240426 13:20:36.997118 28731 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.044857 29013 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.389s
I20240426 13:20:37.196689 29051 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:37.197170 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:37.197413 29026 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20240426 13:20:37.197851 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.198324 28999 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:37.198055 29023 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20240426 13:20:37.198838 29005 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:37.165613 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:37.214368 27911 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.270009 25104 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:37.224300 28062 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:37.277343 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.217777 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:37.180892 28659 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.164651 29025 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:37.224618 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 257927us
I20240426 13:20:37.251483 28999 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.242087 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.037261 29034 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.178s
I20240426 13:20:37.247771 25026 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:37.216325 28927 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.254650 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:37.257611 29008 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.270923 29005 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.269659 27911 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.214797 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:37.287042 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.010s	sys 0.014s
W20240426 13:20:37.352877 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.010s	sys 0.014s
W20240426 13:20:37.353269 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 656478us
I20240426 13:20:37.354068 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:37.287473 25624 raft_consensus.cc:2396] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240426 13:20:37.256884 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.304136 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.317128 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:37.225381 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:37.263803 28937 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.783s	user 0.012s	sys 0.001s
I20240426 13:20:37.340101 25030 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:37.346272 28999 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:37.222434 28735 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:37.335702 28753 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:37.350610 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:37.300391 28659 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:37.356384 25096 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:37.354087 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 139385us
I20240426 13:20:37.365109 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.342110 29009 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.183477 29018 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.358021 29005 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:37.415203 25025 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:37.300809 29017 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.338259 29050 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:37.366082 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49060.6us
I20240426 13:20:37.347987 28903 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:37.423640 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.429952 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.430385 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:37.353451 28062 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:37.427011 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:37.364949 28900 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:37.443043 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:37.431342 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:37.430606 25105 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:37.450464 25025 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:37.438606 24951 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.535s	user 0.009s	sys 0.002s
W20240426 13:20:37.454288 28855 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.005s	sys 0.009s
W20240426 13:20:37.458644 28855 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.005s	sys 0.011s
W20240426 13:20:37.432148 29053 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20240426 13:20:37.405969 28927 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:37.457957 24951 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
W20240426 13:20:37.476280 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 122324us
W20240426 13:20:37.513860 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70932.1us
I20240426 13:20:37.426652 29047 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:37.449790 28903 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.335948 29025 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.532820 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:37.534252 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:37.360597 29008 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.449944 28900 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.415424 28809 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:37.374907 28937 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.520727 28733 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.463646 28753 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:37.465533 29017 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.550352 29006 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:37.561916 29027 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:37.562443 29035 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.562820 29037 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:37.564426 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:37.523417 29045 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:37.571542 25624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:37.600235 25624 raft_consensus.cc:595] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:37.600776 25624 raft_consensus.cc:603] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:37.534369 28927 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:37.605728 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 202970us
W20240426 13:20:37.560303 29032 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
W20240426 13:20:37.574657 28998 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.689s	user 0.004s	sys 0.005s
W20240426 13:20:37.571534 29042 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20240426 13:20:37.579208 28822 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:37.593446 25022 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:37.551064 28900 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:37.596858 24951 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:37.584244 28733 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.502653 28855 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:37.565023 28660 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.027s	user 0.011s	sys 0.000s
I20240426 13:20:37.593312 29027 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.638567 28927 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.583400 29017 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.582979 29010 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20240426 13:20:37.599489 29035 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.651420 29051 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:37.598526 24956 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.001s
I20240426 13:20:37.546283 28903 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.594143 28745 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.680394 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:37.659000 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:37.688295 25027 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.006s	sys 0.036s
W20240426 13:20:37.693727 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.009s	sys 0.005s
W20240426 13:20:37.732196 25027 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.006s	sys 0.037s
W20240426 13:20:37.732277 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.010s	sys 0.005s
W20240426 13:20:37.733246 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 413821us
I20240426 13:20:37.721954 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:37.741405 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 375450us
I20240426 13:20:37.730830 24956 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:37.557825 29077 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.139s
W20240426 13:20:37.746476 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66129us
I20240426 13:20:37.676977 29034 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.745664 25096 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:37.747128 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:37.659535 28822 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:37.761688 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102774us
I20240426 13:20:37.763198 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:37.754228 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.000s	sys 0.011s
I20240426 13:20:37.754559 24951 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:37.754948 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:37.749912 28898 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.015s	sys 0.039s
W20240426 13:20:37.765543 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.000s	sys 0.012s
I20240426 13:20:37.541635 29047 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.442656 29050 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.722587 28937 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:37.767946 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 694794us
I20240426 13:20:37.604318 28753 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.647684 28999 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.811744 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:37.805491 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51396.5us
I20240426 13:20:37.813777 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:37.821425 28997 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.836s	user 0.004s	sys 0.006s
I20240426 13:20:37.772800 29051 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:37.824179 25024 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:37.824358 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:37.764190 28927 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 13:20:37.836263 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 302135us
I20240426 13:20:37.837613 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:37.838380 25105 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:37.822453 29018 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:37.840210 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:37.825273 24951 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20240426 13:20:37.778317 28898 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.016s	sys 0.039s
I20240426 13:20:37.806876 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:37.658550 28998 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.682937 28733 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.728487 28660 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.887027 25102 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 13:20:37.629199 28062 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.882593 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:37.883569 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69885us
I20240426 13:20:37.854254 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:37.886906 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46787.8us
W20240426 13:20:37.942698 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60205.2us
W20240426 13:20:37.943315 25589 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.540s (SENT)
I20240426 13:20:37.943548 25588 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:37.943727 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:37.803189 29065 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.836835 28822 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:37.839881 28937 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:37.945783 25099 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:37.746409 28745 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.944316 25624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3014c46944bc4cbb82624ac909221634"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:37.832437 29051 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:37.845393 29010 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:37.846832 28997 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:37.986088 25026 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:37.962945 25624 raft_consensus.cc:595] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:37.986881 25624 raft_consensus.cc:603] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:37.843557 28927 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:37.942217 29053 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20240426 13:20:38.003600 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 160051us
I20240426 13:20:37.942102 29018 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:38.012926 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 122576us
I20240426 13:20:38.022558 25101 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:37.773612 29034 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:37.971130 29051 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:37.961263 29009 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:37.961500 28937 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:37.884159 29058 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:37.997350 28927 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:38.052832 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:37.972765 29010 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:37.839741 29045 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.174s
I20240426 13:20:37.839419 29005 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.092527 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:38.079244 28937 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:38.079831 29018 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:38.122156 29046 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.321s	user 0.000s	sys 0.001s
I20240426 13:20:38.122354 29018 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.122715 28808 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:38.114406 28680 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:38.123245 29073 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.344s	user 0.000s	sys 0.001s
I20240426 13:20:38.123035 29018 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.123636 29010 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.123023 29030 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:38.125505 29010 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:38.125285 29030 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.126842 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2324) took 1245 ms. Trace:
I20240426 13:20:38.127452 29030 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:38.122625 29079 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:38.128604 29079 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:38.128403 28680 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.128988 29079 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:38.130658 28680 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.132441 28822 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:38.132439 29018 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.136833 28660 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.137100 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2061) took 1112 ms. Trace:
I20240426 13:20:38.137537 25104 rpcz_store.cc:276] 0426 13:20:37.035052 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.035198 (+   146us) service_pool.cc:225] Handling call
0426 13:20:37.035510 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:37.276292 (+240782us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:38.136881 (+860589us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:38.137046 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":538645120}
I20240426 13:20:38.138041 28898 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.033469 29084 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:38.033502 29064 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.159996 29064 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.161015 28733 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.162144 29064 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.162550 29011 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:38.163002 29011 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:38.164413 28998 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.127231 24948 rpcz_store.cc:276] 0426 13:20:36.881632 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.898578 (+ 16946us) service_pool.cc:225] Handling call
0426 13:20:36.898897 (+   319us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:36.983137 (+ 84240us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:37.005776 (+ 22639us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.444697 (+438921us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.526677 (+ 81980us) log.cc:842] Serialized 26 byte log entry
0426 13:20:37.526911 (+   234us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:37.529748 (+  2837us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.576795 (+ 47047us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b18003e7280. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:37.843272 (+266477us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.843361 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:38.126611 (+283250us) raft_consensus.cc:1671] finished
0426 13:20:38.126633 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:38.126780 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69432,"prepare.queue_time_us":73365,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":69,"raft.queue_time_us":864569,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":54502,"spinlock_wait_cycles":62297856,"thread_start_us":580317,"threads_started":3,"wal-append.queue_time_us":864604}
I20240426 13:20:38.166646 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 525) took 1120 ms. Trace:
I20240426 13:20:38.166988 25022 rpcz_store.cc:276] 0426 13:20:37.046572 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.294278 (+247706us) service_pool.cc:225] Handling call
0426 13:20:38.166434 (+872156us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:38.166564 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":524941184}
I20240426 13:20:38.092602 29058 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.154309 29043 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:38.155007 28753 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.156561 29002 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:38.158093 29005 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:37.984148 28855 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.159121 28903 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.163863 29011 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.169852 24956 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:38.052573 29061 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.176347 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:38.123176 28808 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:38.078446 29077 raft_consensus.cc:2741] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:20:38.191792 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:38.262462 25030 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:37.960726 29066 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.198s
I20240426 13:20:38.081830 29025 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.100605 29009 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:38.293704 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 350208us
W20240426 13:20:38.243566 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 300546us
W20240426 13:20:38.251892 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 199153us
W20240426 13:20:38.256605 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 244474us
I20240426 13:20:37.983783 29065 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.238867 28906 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.316413 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:38.197767 29008 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.138597 28898 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.301329 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:38.276496 28716 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.239s	user 0.007s	sys 0.007s
I20240426 13:20:38.281433 25615 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:38.247004 28998 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:38.247390 28753 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.314989 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:38.114842 29072 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.448s
I20240426 13:20:38.247678 29002 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:38.197458 28680 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20240426 13:20:38.293118 29011 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:38.361019 24953 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.027s	sys 0.048s
I20240426 13:20:38.363023 25621 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:38.383399 24953 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.028s	sys 0.048s
W20240426 13:20:38.364353 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63213.8us
I20240426 13:20:38.304539 29017 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.344049 28999 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:38.343050 28626 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.531s	user 0.013s	sys 0.000s
I20240426 13:20:38.366070 24955 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:38.367141 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:38.354112 29064 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:38.360358 29045 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.365839 29077 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20240426 13:20:38.381145 28998 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:38.331355 28997 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:38.335247 28943 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.019s	sys 0.030s
I20240426 13:20:38.390895 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.391587 25102 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20240426 13:20:38.392156 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.344933 29035 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:38.392931 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 449619us
I20240426 13:20:38.393862 25104 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:20:38.421741 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:38.422577 25096 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:38.407954 25615 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:38.423230 25615 raft_consensus.cc:595] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:38.423703 25615 raft_consensus.cc:603] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:38.423723 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:38.394191 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:38.396236 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:38.425216 25099 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:38.396564 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:38.368945 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.343827 29027 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:38.253701 29084 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20240426 13:20:38.406035 29063 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.406304 28745 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:38.397310 29050 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:38.344081 28716 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.319s	user 0.008s	sys 0.007s
I20240426 13:20:38.380719 29018 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:38.415938 25100 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:38.390779 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.370707 29047 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.343309 28906 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.435992 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.499485 25102 raft_consensus.cc:2396] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20240426 13:20:38.264593 29061 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:38.406661 29052 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.006s	sys 0.010s
I20240426 13:20:38.449693 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:38.464555 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70457.4us
I20240426 13:20:38.509492 25098 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:38.510385 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:38.399680 29064 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:38.421147 28943 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.019s	sys 0.031s
I20240426 13:20:38.516785 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.518525 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.492851 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:38.519248 25102 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:38.398097 28822 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20240426 13:20:38.495833 28735 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.087s	user 0.008s	sys 0.000s
I20240426 13:20:38.413174 28999 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20240426 13:20:38.413424 28998 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:38.497763 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.474387 28716 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:38.507937 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.486269 29035 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:38.520879 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:38.568508 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47717.7us
I20240426 13:20:38.463275 29034 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.403205 28626 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.541633 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:38.391453 28753 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.542321 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2448) took 1925 ms. Trace:
I20240426 13:20:38.406095 29009 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.557464 25104 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:20:38.519944 29052 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.006s	sys 0.011s
I20240426 13:20:38.511577 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:38.564153 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:38.495719 28659 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.046s	user 0.008s	sys 0.001s
I20240426 13:20:38.545879 25100 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:38.569856 25099 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:38.583817 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:38.588311 29104 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:38.588443 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.588973 29107 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:38.607723 25102 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:38.607957 25096 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:38.589304 29034 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:38.564262 28943 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.532915 29093 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.391s
I20240426 13:20:38.507963 29047 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.516852 29058 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:38.413007 29045 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.370755 28808 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.499325 28745 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.102s
I20240426 13:20:38.498677 29018 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:38.531848 29072 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.194s
I20240426 13:20:38.628804 25098 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:38.735350 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:38.595890 25031 rpcz_store.cc:276] 0426 13:20:36.616637 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.642154 (+ 25517us) service_pool.cc:225] Handling call
0426 13:20:36.665994 (+ 23840us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:36.706386 (+ 40392us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:36.706423 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:36.715847 (+  9424us) log.cc:842] Serialized 26 byte log entry
0426 13:20:36.947771 (+231924us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:36.948124 (+   353us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.413040 (+464916us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.413125 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:38.494371 (+1081246us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b1800437b00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:38.497454 (+  3083us) raft_consensus.cc:1671] finished
0426 13:20:38.542051 (+ 44597us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:38.542243 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37600,"prepare.queue_time_us":60063,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":284059,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":6622,"spinlock_wait_cycles":7047936,"thread_start_us":655717,"threads_started":2,"wal-append.queue_time_us":428222}
I20240426 13:20:38.722965 25099 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:38.542948 29099 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.564947 28998 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:39.111505 25102 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:38.559011 29008 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:39.065905 28707 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.170s	user 0.013s	sys 0.002s
W20240426 13:20:39.092114 29048 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.478s	user 0.001s	sys 0.001s
W20240426 13:20:39.139053 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 675386us
I20240426 13:20:38.602017 29077 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.091115 29033 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.958166 29117 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:38.981487 28999 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20240426 13:20:39.130277 29051 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.029s	user 0.009s	sys 0.000s
I20240426 13:20:38.557067 29079 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:39.122035 28943 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.162782 29024 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:38.556495 28735 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.133420 29035 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:39.187544 29098 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:39.188486 29106 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:39.189940 29106 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:39.190253 28927 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.020s	user 0.005s	sys 0.004s
I20240426 13:20:39.190783 28927 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:39.192267 29075 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
I20240426 13:20:38.647114 28659 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.194423 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2451) took 2506 ms. Trace:
I20240426 13:20:39.135351 29065 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.197578 25623 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20240426 13:20:39.122192 29047 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.993808 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1244) took 2393 ms. Trace:
I20240426 13:20:39.221046 25103 rpcz_store.cc:276] 0426 13:20:36.577611 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.577772 (+   161us) service_pool.cc:225] Handling call
0426 13:20:38.970367 (+2392595us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:38.971076 (+   709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572972288}
I20240426 13:20:39.164517 28999 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:39.161764 29117 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:39.206138 28927 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:39.192174 29055 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.713s	user 0.005s	sys 0.000s
I20240426 13:20:39.194806 25029 rpcz_store.cc:276] 0426 13:20:36.688588 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.932111 (+243523us) service_pool.cc:225] Handling call
0426 13:20:36.951765 (+ 19654us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:37.026673 (+ 74908us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:37.092531 (+ 65858us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.412915 (+320384us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.450482 (+ 37567us) log.cc:842] Serialized 26 byte log entry
0426 13:20:37.450624 (+   142us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:37.450652 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.450717 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.191592 (+1740875us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:39.194126 (+  2534us) raft_consensus.cc:1671] finished
0426 13:20:39.194225 (+    99us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.194369 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5345,"prepare.queue_time_us":167941,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":238477,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":17530,"spinlock_wait_cycles":29447040,"thread_start_us":233777,"threads_started":2,"wal-append.queue_time_us":1399970}
I20240426 13:20:39.244163 24948 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20240426 13:20:39.231298 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 809587us
I20240426 13:20:38.694078 28997 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:39.301990 25621 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.912s	user 0.010s	sys 0.000s
I20240426 13:20:39.141754 29008 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.313573 25621 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:39.315186 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 767) took 1053 ms. Trace:
W20240426 13:20:38.970189 29112 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20240426 13:20:38.881515 29037 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.288s	user 0.010s	sys 0.000s
I20240426 13:20:38.762863 28822 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:39.316721 25621 rpcz_store.cc:276] 0426 13:20:38.261289 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.261658 (+   369us) service_pool.cc:225] Handling call
0426 13:20:39.315000 (+1053342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:39.257843 28999 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:39.208410 29065 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.777194 29027 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.994252 29066 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:38.794104 29113 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.381981 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:39.261214 28626 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.309478 28903 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.375443 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
W20240426 13:20:39.117727 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 606211us
I20240426 13:20:39.332226 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:39.426865 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 94759.7us
I20240426 13:20:39.427704 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:39.181016 29051 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.431555 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:39.152377 28707 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.356623 29037 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:39.333665 28753 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:39.397503 25022 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.007s	sys 0.007s
I20240426 13:20:39.424736 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2450) took 2759 ms. Trace:
W20240426 13:20:39.425707 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 266020us
I20240426 13:20:39.395256 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:39.364974 28822 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.356997 29025 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:39.437181 25022 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.991s	user 0.008s	sys 0.007s
I20240426 13:20:39.368552 29061 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:39.364209 29104 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:39.437386 25023 rpcz_store.cc:276] 0426 13:20:36.664990 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.716881 (+ 51891us) service_pool.cc:225] Handling call
0426 13:20:36.733313 (+ 16432us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:36.749521 (+ 16208us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:36.749606 (+    85us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.886703 (+1137097us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:20:37.949516 (+ 62813us) log.cc:842] Serialized 26 byte log entry
0426 13:20:38.248790 (+299274us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:38.248887 (+    97us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:38.249089 (+   202us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.384915 (+1135826us) raft_consensus.cc:1671] finished
0426 13:20:39.384942 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.424676 (+ 39734us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145274,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":137556736,"thread_start_us":234326,"threads_started":1,"wal-append.queue_time_us":583987}
I20240426 13:20:39.438719 25621 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:39.459154 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 743) took 1015 ms. Trace:
I20240426 13:20:39.459542 25621 rpcz_store.cc:276] 0426 13:20:38.444272 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.394917 (+950645us) service_pool.cc:225] Handling call
0426 13:20:39.459059 (+ 64142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:39.461376 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2449) took 2816 ms. Trace:
I20240426 13:20:38.971895 28716 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.391866 29034 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.473037 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:39.533285 29022 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:39.458920 28927 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:39.536078 29110 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:39.434296 28659 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.480031 29072 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:39.534683 29018 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.535431 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:39.576570 25621 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:39.450218 28898 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.434028 29090 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.440344 28626 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.455804 28822 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:39.461956 25028 rpcz_store.cc:276] 0426 13:20:36.644810 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.645381 (+   571us) service_pool.cc:225] Handling call
0426 13:20:36.652655 (+  7274us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:36.819786 (+167131us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:36.883525 (+ 63739us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.032742 (+149217us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.043144 (+ 10402us) log.cc:842] Serialized 26 byte log entry
0426 13:20:37.043301 (+   157us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:37.043343 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.043429 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.461115 (+2417686us) raft_consensus.cc:1671] finished
0426 13:20:39.461143 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.461308 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34068,"prepare.queue_time_us":59678,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":176577,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":5501,"thread_start_us":235136,"threads_started":2,"wal-append.queue_time_us":950281}
W20240426 13:20:39.555199 28846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.010s	sys 0.033s
I20240426 13:20:39.484844 25025 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20240426 13:20:39.567826 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:39.481374 29129 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.202s	user 0.000s	sys 0.004s
I20240426 13:20:39.381577 29065 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.566613 29061 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.533396 29036 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:39.546031 29037 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:39.397564 29027 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.571391 29022 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:39.588629 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:39.576447 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.002s	sys 0.006s
W20240426 13:20:39.566691 29069 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.739s	user 0.001s	sys 0.001s
I20240426 13:20:39.696308 25621 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:39.440986 28903 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.460441 29025 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.454866 29063 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:38.566323 29002 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.190366 29106 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.159258 29077 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.594103 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2320) took 2886 ms. Trace:
W20240426 13:20:39.633797 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 67376.6us
I20240426 13:20:39.566049 28753 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.577999 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:39.164273 29033 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.424639 29066 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:39.638751 29132 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20240426 13:20:39.423113 29113 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.639271 28745 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:39.132274 29099 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.192090 29024 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.731263 25621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:39.690440 28943 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:39.643628 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 216018us
I20240426 13:20:39.460165 29103 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.504s
I20240426 13:20:39.639971 29139 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:39.650306 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.001s	sys 0.009s
W20240426 13:20:39.647470 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.014s	sys 0.030s
I20240426 13:20:39.461305 29035 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.704133 28735 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:39.744621 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 703) took 2105 ms (client timeout 2172 ms). Trace:
I20240426 13:20:39.744756 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:39.745119 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.018s	user 0.015s	sys 0.036s
W20240426 13:20:39.754619 24955 rpcz_store.cc:269] 0426 13:20:37.638878 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.751687 (+112809us) service_pool.cc:225] Handling call
0426 13:20:39.743926 (+1992239us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54003679d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:39.744024 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1181537152}
W20240426 13:20:39.744902 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.102s	user 0.005s	sys 0.011s
I20240426 13:20:39.747259 25621 raft_consensus.cc:595] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:39.761732 25621 raft_consensus.cc:603] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:39.762145 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38842 (request call id 4) took 1255 ms. Trace:
I20240426 13:20:39.762509 25621 rpcz_store.cc:276] 0426 13:20:38.507122 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.706037 (+1198915us) service_pool.cc:225] Handling call
0426 13:20:39.762076 (+ 56039us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:39.636165 28846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.011s	sys 0.033s
W20240426 13:20:39.657155 28900 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.934s	user 0.013s	sys 0.000s
W20240426 13:20:39.651028 29031 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.299s	user 0.000s	sys 0.000s
W20240426 13:20:39.650175 29088 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20240426 13:20:39.648267 29131 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.260s	user 0.000s	sys 0.000s
W20240426 13:20:39.647578 29001 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.932s	user 0.001s	sys 0.003s
W20240426 13:20:39.648811 29127 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:20:39.588949 29130 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.626556 29100 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.723789 29022 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.777956 29008 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:39.647087 29114 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.396936 29105 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.541s
I20240426 13:20:39.720182 24949 rpcz_store.cc:276] 0426 13:20:36.701502 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.701689 (+   187us) service_pool.cc:225] Handling call
0426 13:20:36.702124 (+   435us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:36.940471 (+238347us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:36.940500 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:36.962793 (+ 22293us) log.cc:842] Serialized 52 byte log entry
0426 13:20:37.185008 (+222215us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:37.430453 (+245445us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.659565 (+229112us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.575713 (+916148us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:39.200751 (+625038us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b18003d1440. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:39.367862 (+167111us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.368106 (+   244us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.587891 (+219785us) raft_consensus.cc:1671] finished
0426 13:20:39.587919 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.588127 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50967,"prepare.queue_time_us":392322,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":68,"raft.queue_time_us":1012633,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":31830,"spinlock_wait_cycles":370259584,"thread_start_us":993225,"threads_started":5,"wal-append.queue_time_us":306937}
I20240426 13:20:39.643918 29079 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:39.642391 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 210995us
I20240426 13:20:39.626103 28626 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:39.740550 29119 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.335s	user 0.001s	sys 0.001s
I20240426 13:20:39.740978 29102 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
W20240426 13:20:39.741559 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 268603us
I20240426 13:20:39.591785 29117 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.751003 29139 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:39.696346 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s	user 0.002s	sys 0.006s
I20240426 13:20:39.730448 29078 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:39.727690 28429 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.589470 29034 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.811563 29024 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:39.769903 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.032s	user 0.021s	sys 0.046s
W20240426 13:20:39.766341 25589 connection.cc:232] RPC call timeout handler was delayed by -1.545633566999868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
I20240426 13:20:39.727084 29047 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:39.737057 28745 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:39.774472 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:39.774994 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.695289 29061 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.827795 29139 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.829071 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2321) took 3103 ms. Trace:
I20240426 13:20:39.829489 29061 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.781461 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:39.830396 29077 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:39.831153 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2444) took 3258 ms. Trace:
I20240426 13:20:39.831498 29106 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.832119 29066 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.832329 29025 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.833252 29063 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.797693 29114 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.596778 29090 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.835742 29114 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.768221 28659 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:39.647282 29081 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.316s
I20240426 13:20:39.765166 29018 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:39.787070 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:39.787288 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 537) took 1759 ms. Trace:
I20240426 13:20:39.864161 25022 rpcz_store.cc:276] 0426 13:20:38.027989 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.283662 (+255673us) service_pool.cc:225] Handling call
0426 13:20:38.284122 (+   460us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:38.350140 (+ 66018us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:38.350171 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:38.361501 (+ 11330us) log.cc:842] Serialized 26 byte log entry
0426 13:20:39.443847 (+1082346us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:39.443902 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.443992 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.787091 (+343099us) raft_consensus.cc:1671] finished
0426 13:20:39.787111 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.787227 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65574,"raft.queue_time_us":69557,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":12401,"thread_start_us":997173,"threads_started":1,"wal-append.queue_time_us":1303869}
I20240426 13:20:39.730129 28753 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.724419 29045 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.584005 29104 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.794984 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:39.796957 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 219079us
I20240426 13:20:39.783661 28707 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.797463 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.786437 29051 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.873085 25103 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:39.803050 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 697) took 2938 ms. Trace:
I20240426 13:20:39.773905 28900 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.797891 29022 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:39.742236 29145 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:39.805552 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:39.806957 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 753) took 4031 ms. Trace:
I20240426 13:20:39.807910 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 737) took 2630 ms. Trace:
I20240426 13:20:39.744601 29111 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
W20240426 13:20:39.814229 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.021s	sys 0.046s
I20240426 13:20:39.811477 28429 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.815187 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:39.815187 28917 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:39.790868 29130 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.818572 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2323) took 2985 ms. Trace:
W20240426 13:20:39.819427 29011 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.430s	user 0.010s	sys 0.000s
I20240426 13:20:39.819742 28870 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:39.820147 28937 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.640s	user 0.009s	sys 0.003s
I20240426 13:20:39.824378 29117 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:39.814967 29078 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:39.825809 29125 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20240426 13:20:39.915920 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:39.827487 28903 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.794981 29100 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.831503 25027 rpcz_store.cc:276] 0426 13:20:36.572710 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.573521 (+   811us) service_pool.cc:225] Handling call
0426 13:20:36.628202 (+ 54681us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:36.632382 (+  4180us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:36.632414 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:36.653460 (+ 21046us) log.cc:842] Serialized 52 byte log entry
0426 13:20:36.857314 (+203854us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:37.732649 (+875335us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.762725 (+ 30076us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.762806 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.830932 (+2068126us) raft_consensus.cc:1671] finished
0426 13:20:39.830950 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.831090 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42235,"prepare.queue_time_us":63706,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":40470,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":40113,"thread_start_us":987634,"threads_started":2,"wal-append.queue_time_us":2534409}
I20240426 13:20:39.836284 28659 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:39.922369 28659 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:39.923424 28900 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.863173 25102 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:39.829455 24952 rpcz_store.cc:276] 0426 13:20:36.725274 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.725788 (+   514us) service_pool.cc:225] Handling call
0426 13:20:36.760393 (+ 34605us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:36.860913 (+100520us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:36.883966 (+ 23053us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.882517 (+998551us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.883042 (+   525us) log.cc:842] Serialized 26 byte log entry
0426 13:20:37.884305 (+  1263us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:37.884753 (+   448us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.947877 (+ 63124us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.947953 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.649416 (+1701463us) spinlock_profiling.cc:243] Waited 7.42 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:39.828873 (+179457us) raft_consensus.cc:1671] finished
0426 13:20:39.828896 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.829009 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114528,"prepare.queue_time_us":31727,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":14145,"raft.queue_time_us":875798,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":38472,"spinlock_wait_cycles":17069440,"thread_start_us":932598,"threads_started":2,"wal-append.queue_time_us":2477633}
I20240426 13:20:39.808699 29140 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:39.811590 29033 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:39.837669 28745 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:39.929347 29140 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:39.876252 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:39.862365 29034 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:39.939393 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63239.9us
W20240426 13:20:39.867372 29127 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240426 13:20:39.874894 24947 rpcz_store.cc:276] 0426 13:20:36.864412 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.889989 (+ 25577us) service_pool.cc:225] Handling call
0426 13:20:36.916507 (+ 26518us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:37.213194 (+296687us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:37.213218 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.213687 (+   469us) log.cc:842] Serialized 26 byte log entry
0426 13:20:37.391306 (+177619us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:37.391364 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.799683 (+2408319us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54000bf754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:39.799725 (+    42us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.799767 (+    42us) raft_consensus.cc:1671] finished
0426 13:20:39.799781 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.803001 (+  3220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7043,"raft.queue_time_us":310592,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":22175,"spinlock_wait_cycles":681560960,"thread_start_us":127854,"threads_started":1,"wal-append.queue_time_us":429956,"wal-append.run_cpu_time_us":2802,"wal-append.run_wall_time_us":1446406}
W20240426 13:20:39.882784 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77313.4us
I20240426 13:20:39.879596 25618 rpcz_store.cc:276] 0426 13:20:37.173091 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.173245 (+   154us) service_pool.cc:225] Handling call
0426 13:20:37.195292 (+ 22047us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:37.452999 (+257707us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:37.453030 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:37.453073 (+    43us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:37.453112 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.802991 (+2349879us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54005c1054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:39.803034 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.803133 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592239872}
I20240426 13:20:39.879503 25617 rpcz_store.cc:276] 0426 13:20:35.775059 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.775212 (+   153us) service_pool.cc:225] Handling call
0426 13:20:35.794995 (+ 19783us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:35.803550 (+  8555us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:35.803593 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:35.809640 (+  6047us) log.cc:842] Serialized 26 byte log entry
0426 13:20:36.360587 (+550947us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:36.360640 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:36.360747 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.646805 (+3286058us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:39.799552 (+152747us) raft_consensus.cc:1671] finished
0426 13:20:39.799574 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.805966 (+  6392us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":953,"raft.queue_time_us":1081,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":19602,"spinlock_wait_cycles":47270144,"thread_start_us":550572,"threads_started":1,"wal-append.queue_time_us":848372}
I20240426 13:20:39.883656 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:39.817518 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 704) took 1521 ms (client timeout 1539 ms). Trace:
W20240426 13:20:39.942829 24955 rpcz_store.cc:269] 0426 13:20:38.295870 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.761184 (+1465314us) service_pool.cc:225] Handling call
0426 13:20:39.817468 (+ 56284us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:39.891764 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54359.9us
I20240426 13:20:39.869169 29065 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.897904 25624 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:39.946890 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:39.947378 29100 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.902132 28917 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.948269 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:39.948477 28906 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:39.949311 25617 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:39.949776 28917 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.951184 28429 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:39.951468 29137 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.416s	user 0.001s	sys 0.000s
I20240426 13:20:39.952689 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2329) took 2919 ms. Trace:
I20240426 13:20:39.955111 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.956566 29064 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:39.958006 29006 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:39.958490 29006 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:39.953110 24951 rpcz_store.cc:276] 0426 13:20:37.033226 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.841706 (+808480us) service_pool.cc:225] Handling call
0426 13:20:37.943654 (+101948us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:38.202820 (+259166us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:38.202853 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:38.252170 (+ 49317us) log.cc:842] Serialized 52 byte log entry
0426 13:20:38.513510 (+261340us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:38.560238 (+ 46728us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.742997 (+182759us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:38.743101 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.952449 (+1209348us) raft_consensus.cc:1671] finished
0426 13:20:39.952473 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.952622 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":373397,"prepare.queue_time_us":46325,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":14177,"raft.queue_time_us":1140909,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":12237,"thread_start_us":372117,"threads_started":2,"wal-append.queue_time_us":1113169}
I20240426 13:20:39.899993 25621 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20240426 13:20:39.949832 25617 raft_consensus.cc:595] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:39.896049 24953 rpcz_store.cc:276] 0426 13:20:36.832954 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.872798 (+ 39844us) service_pool.cc:225] Handling call
0426 13:20:36.874699 (+  1901us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:36.994594 (+119895us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:36.994655 (+    61us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:36.995358 (+   703us) log.cc:842] Serialized 52 byte log entry
0426 13:20:37.440289 (+444931us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:37.732185 (+291896us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.408673 (+676488us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.408833 (+   160us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:38.408900 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:39.818411 (+1409511us) raft_consensus.cc:1671] finished
0426 13:20:39.818425 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:39.818532 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91661,"prepare.queue_time_us":173893,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":1307458,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":94383,"thread_start_us":1291274,"threads_started":3,"wal-append.queue_time_us":1328947}
I20240426 13:20:39.965632 25617 raft_consensus.cc:603] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:39.868837 29102 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:39.902362 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 740) took 1462 ms. Trace:
I20240426 13:20:39.903707 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:39.967176 25105 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:39.904067 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:39.904789 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:39.907263 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94956.3us
I20240426 13:20:39.879115 29145 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:39.910197 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:39.913899 29149 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20240426 13:20:39.826294 29099 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.899161 29083 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:39.868712 29018 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.918386 25029 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:39.894492 29027 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.901603 29051 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.918772 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:39.984858 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:39.905431 28707 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:39.906385 29128 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20240426 13:20:39.903232 29009 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:39.926643 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.905380 29126 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:39.906359 28937 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:39.930197 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:39.798947 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:39.932819 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 117718us
I20240426 13:20:39.916100 28626 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:39.944851 25070 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.696s (SENT) [suppressed 9 similar messages]
I20240426 13:20:39.968713 25022 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:39.968151 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64176.3us
I20240426 13:20:39.966394 25623 rpcz_store.cc:276] 0426 13:20:38.440235 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.440377 (+   142us) service_pool.cc:225] Handling call
0426 13:20:38.484504 (+ 44127us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.198065 (+713561us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.899522 (+701457us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:39.902297 (+  2775us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1631778176}
I20240426 13:20:39.970983 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:39.973292 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.926000 29113 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:39.974572 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:39.976703 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.984541 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:39.945477 29111 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:39.942648 29140 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:39.987144 25618 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:39.987844 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:39.989506 25617 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:20:39.972683 29145 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:39.972764 28997 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:20:40.124998 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 180184us
I20240426 13:20:40.125270 24952 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:40.126463 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:40.138473 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 768) took 1547 ms. Trace:
I20240426 13:20:40.138628 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:40.139765 25102 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:40.139361 25617 rpcz_store.cc:276] 0426 13:20:38.591548 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.975747 (+1384199us) service_pool.cc:225] Handling call
0426 13:20:39.989250 (+ 13503us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:40.098106 (+108856us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.138412 (+ 40306us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:40.129874 29043 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.873s	user 0.010s	sys 0.001s
I20240426 13:20:40.129391 29102 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.130362 29146 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20240426 13:20:40.130587 29037 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:40.148874 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 769) took 1049 ms. Trace:
W20240426 13:20:40.121217 29128 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:20:40.131634 29078 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.150599 28997 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:40.041328 29083 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:40.169500 28997 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:40.170095 29083 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:40.132509 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2340) took 1681 ms (client timeout 1710 ms). Trace:
W20240426 13:20:40.172255 24947 rpcz_store.cc:269] 0426 13:20:38.450837 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.941672 (+1490835us) service_pool.cc:225] Handling call
0426 13:20:40.132447 (+190775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:40.173218 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.174042 24947 raft_consensus.cc:2396] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20240426 13:20:40.174418 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 706) took 1782 ms (client timeout 2247 ms). Trace:
W20240426 13:20:40.127900 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2343) took 1676 ms (client timeout 1798 ms). Trace:
W20240426 13:20:40.174747 24947 rpcz_store.cc:269] 0426 13:20:38.392125 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.172974 (+1780849us) service_pool.cc:225] Handling call
0426 13:20:40.174372 (+  1398us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:40.174875 24951 rpcz_store.cc:269] 0426 13:20:38.451592 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.964860 (+1513268us) service_pool.cc:225] Handling call
0426 13:20:40.127837 (+162977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:40.175768 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.176102 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.176321 24947 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:40.176878 24951 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:40.177767 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.125171 24955 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:40.136356 25098 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:40.125026 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 326151us
I20240426 13:20:40.318706 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.135852 25105 raft_consensus.cc:2396] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20240426 13:20:40.145359 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2344) took 1693 ms (client timeout 2104 ms). Trace:
W20240426 13:20:40.319738 24953 rpcz_store.cc:269] 0426 13:20:38.451857 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.966064 (+1514207us) service_pool.cc:225] Handling call
0426 13:20:40.145307 (+179243us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:40.147187 28927 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.568s	user 0.006s	sys 0.001s
I20240426 13:20:40.323104 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.324242 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.324443 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.325287 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:40.060597 29161 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20240426 13:20:40.063431 28998 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.924s	user 0.006s	sys 0.003s
I20240426 13:20:40.332867 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.149283 25623 rpcz_store.cc:276] 0426 13:20:39.099493 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.148103 (+1048610us) service_pool.cc:225] Handling call
0426 13:20:40.148403 (+   300us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:40.148578 (+   175us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:40.148617 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:40.148655 (+    38us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:40.148681 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.148726 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.148825 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:40.153375 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:40.153587 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2341) took 1702 ms (client timeout 1696 ms). Trace:
I20240426 13:20:40.187945 29111 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:40.162024 29124 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.084904 28753 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.162379 29043 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:39.994488 29051 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:40.333851 29158 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20240426 13:20:40.041880 28707 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.163311 28937 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.136607 29035 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:40.397572 29035 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:40.398027 29035 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:40.399072 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2077) took 1892 ms. Trace:
I20240426 13:20:40.160809 28626 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:40.064538 28659 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:40.133462 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 162537us
I20240426 13:20:40.150090 29126 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.419509 28937 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.257700 25102 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:20:40.179397 25030 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.261478 25618 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:39.987334 29090 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.187750 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:40.420599 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95360.3us
I20240426 13:20:40.421274 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.331748 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:40.422376 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 707) took 1053 ms. Trace:
I20240426 13:20:40.422678 24947 rpcz_store.cc:276] 0426 13:20:39.369410 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.175557 (+806147us) service_pool.cc:225] Handling call
0426 13:20:40.422329 (+246772us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:40.334188 24952 rpcz_store.cc:269] 0426 13:20:38.451067 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.926163 (+1475096us) service_pool.cc:225] Handling call
0426 13:20:40.153503 (+227340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:40.061759 28758 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.424342 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:40.335240 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:40.338315 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 197070us
W20240426 13:20:40.339249 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 185943us
I20240426 13:20:40.333544 24951 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:40.425590 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2328) took 3428 ms. Trace:
I20240426 13:20:40.366101 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7408112bace482693c999f678aa9d73"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:40.426556 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:40.427192 25098 raft_consensus.cc:595] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:40.427623 25098 raft_consensus.cc:603] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:40.370072 29009 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:40.377980 27911 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:40.381695 29164 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:20:40.384052 29050 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:40.178464 28777 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.180s
I20240426 13:20:40.387787 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.441239 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:40.392701 25615 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:40.174607 28745 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.446911 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.338629 29045 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:40.336064 28927 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:40.341342 29111 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:40.337430 29066 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:40.473589 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.341645 29063 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:40.406562 28626 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:40.399529 25096 rpcz_store.cc:276] 0426 13:20:38.506939 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.121854 (+614915us) service_pool.cc:225] Handling call
0426 13:20:39.139506 (+ 17652us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.458085 (+318579us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:39.458131 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:39.537233 (+ 79102us) log.cc:842] Serialized 26 byte log entry
0426 13:20:39.827679 (+290446us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b18003510a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:39.939638 (+111959us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:39.939679 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.939834 (+   155us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:40.398730 (+458896us) raft_consensus.cc:1671] finished
0426 13:20:40.398756 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.398991 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":350164,"raft.queue_time_us":410417,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":14223,"spinlock_wait_cycles":6423424,"thread_start_us":99657,"threads_started":1,"wal-append.queue_time_us":681275}
W20240426 13:20:40.407029 29167 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20240426 13:20:40.421350 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 233720us
W20240426 13:20:40.248245 29166 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20240426 13:20:40.422147 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:40.163079 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.428864 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.428997 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.431759 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.486953 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:20:40.489267 24947 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:40.426004 24950 rpcz_store.cc:276] 0426 13:20:36.996923 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.809874 (+812951us) service_pool.cc:225] Handling call
0426 13:20:37.845088 (+ 35214us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:38.083093 (+238005us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:38.147626 (+ 64533us) log.cc:842] Serialized 14 byte log entry
0426 13:20:39.358312 (+1210686us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:39.358666 (+   354us) log.cc:842] Serialized 26 byte log entry
0426 13:20:39.358752 (+    86us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:39.358769 (+    17us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.358814 (+    45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:40.425385 (+1066571us) raft_consensus.cc:1671] finished
0426 13:20:40.425405 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.425536 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":547544,"prepare.queue_time_us":71603,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":1396236,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":201,"thread_start_us":737945,"threads_started":3,"wal-append.queue_time_us":1577562}
I20240426 13:20:40.435113 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.435679 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.450678 29045 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:40.436280 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.436280 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.495432 24953 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:40.424182 29043 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:40.440697 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.440542 28917 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:40.340634 28998 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.444159 29050 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:40.336581 29006 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.422613 29130 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.472959 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.436702 29090 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:40.420174 28937 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.475121 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:40.477466 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.536577 25098 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:40.537382 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.538457 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.449960 28900 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:40.482389 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:40.543126 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60801.6us
I20240426 13:20:40.543967 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.545900 25617 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:40.484192 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.490510 25028 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:40.435793 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:40.491617 25022 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:40.476707 29087 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.887s	user 0.000s	sys 0.003s
I20240426 13:20:40.488749 24952 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:40.433357 28758 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.490828 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:20:40.476119 29063 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:40.506935 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.507771 28917 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:40.481954 28626 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:40.507794 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.481667 29066 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:40.370155 29124 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:40.568432 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 530) took 3515 ms (client timeout 1893 ms). Trace:
I20240426 13:20:40.532964 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.533622 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.501385 29045 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:40.539983 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.543176 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:40.569036 25024 rpcz_store.cc:269] 0426 13:20:37.047878 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.377077 (+329199us) service_pool.cc:225] Handling call
0426 13:20:40.563151 (+3186074us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54002ae254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:40.563268 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":878898560}
I20240426 13:20:40.494005 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.549811 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 762) took 3108 ms. Trace:
I20240426 13:20:40.550155 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.550853 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.551522 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.552610 28659 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:40.553076 29006 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:40.553331 28822 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.919s	user 0.007s	sys 0.003s
W20240426 13:20:40.561161 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 125455us
I20240426 13:20:40.511884 29050 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:40.562712 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.562435 28927 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
W20240426 13:20:40.564738 29082 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.054s	user 0.002s	sys 0.002s
I20240426 13:20:40.532225 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:40.784905 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 535) took 2759 ms (client timeout 2377 ms). Trace:
I20240426 13:20:40.785075 24950 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:40.566514 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:40.785449 25030 rpcz_store.cc:269] 0426 13:20:38.025835 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.263178 (+237343us) service_pool.cc:225] Handling call
0426 13:20:40.643743 (+2380565us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54004169d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:40.784821 (+141078us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4257656576}
I20240426 13:20:40.786854 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.570259 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:40.580539 24952 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:40.584334 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.573418 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:40.585592 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.542392 29090 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:40.586324 25102 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:40.586277 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:40.798734 25024 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:40.799065 25031 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.562719 29063 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:40.590037 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:40.775286 29140 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.647s	user 0.011s	sys 0.004s
I20240426 13:20:40.567606 28731 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:40.598063 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:40.809314 25100 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:40.615501 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:40.612010 28999 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.231s	user 0.003s	sys 0.008s
I20240426 13:20:40.620926 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.819478 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.612758 29051 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.616813 28707 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.585670 29066 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.833534 29066 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.593374 28822 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.835253 28822 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.583392 29175 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:40.835911 29175 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.838030 29175 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.565793 28753 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:40.637567 28917 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.746913 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 735) took 3596 ms. Trace:
W20240426 13:20:40.748361 24995 connection.cc:232] RPC call timeout handler was delayed by -2.2710917319999453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20240426 13:20:40.841522 24995 connection.cc:232] RPC call timeout handler was delayed by -2.2053917319999528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20600000000000002s for a response
W20240426 13:20:40.841987 24995 connection.cc:232] RPC call timeout handler was delayed by -2.066791731999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20240426 13:20:40.846153 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.747843 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2070) took 2910 ms. Trace:
I20240426 13:20:40.777523 28758 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.779920 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.781263 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:40.782155 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2457) took 3967 ms. Trace:
I20240426 13:20:40.599367 25098 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20240426 13:20:40.747051 29117 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.835s	user 0.013s	sys 0.003s
I20240426 13:20:40.798728 28998 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.578223 25620 rpcz_store.cc:276] 0426 13:20:37.441276 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.441476 (+   200us) service_pool.cc:225] Handling call
0426 13:20:37.513135 (+ 71659us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:37.835735 (+322600us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:37.835804 (+    69us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:37.853874 (+ 18070us) log.cc:842] Serialized 26 byte log entry
0426 13:20:38.457250 (+603376us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:38.457400 (+   150us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.549444 (+2092044us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54006b26d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:40.549495 (+    51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:40.549542 (+    47us) raft_consensus.cc:1671] finished
0426 13:20:40.549555 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.549756 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13166,"raft.queue_time_us":75970,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":75458,"spinlock_wait_cycles":588384128,"thread_start_us":482345,"threads_started":1,"wal-append.queue_time_us":702470}
I20240426 13:20:40.854084 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.801971 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:40.787681 29028 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 3.249s	user 0.007s	sys 0.004s
I20240426 13:20:40.796676 29035 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:40.778278 29118 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.188s	user 0.005s	sys 0.008s
W20240426 13:20:40.779040 28997 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.001s
I20240426 13:20:40.811132 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.800729 28940 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.812765 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2326) took 3869 ms. Trace:
I20240426 13:20:40.809823 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.816529 28937 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:40.877274 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.868060 24954 rpcz_store.cc:276] 0426 13:20:36.943909 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:36.950340 (+  6431us) service_pool.cc:225] Handling call
0426 13:20:36.951667 (+  1327us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:37.019072 (+ 67405us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:37.412106 (+393034us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.072380 (+660274us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:38.092093 (+ 19713us) log.cc:842] Serialized 26 byte log entry
0426 13:20:38.092255 (+   162us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:38.147921 (+ 55666us) log.cc:842] Serialized 14 byte log entry
0426 13:20:38.169364 (+ 21443us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:38.169439 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:40.473162 (+2303723us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:40.812564 (+339402us) raft_consensus.cc:1671] finished
0426 13:20:40.812594 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.812714 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140277,"prepare.queue_time_us":620875,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":671568,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":34970,"spinlock_wait_cycles":9379072,"thread_start_us":868581,"threads_started":3,"wal-append.queue_time_us":1135365}
I20240426 13:20:40.841933 29176 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.772418 28927 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:40.585806 29145 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.823611 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.826982 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.847563 25024 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:40.847051 25101 rpcz_store.cc:276] 0426 13:20:37.837650 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.839506 (+  1856us) service_pool.cc:225] Handling call
0426 13:20:37.941730 (+102224us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:38.023020 (+ 81290us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.624938 (+2601918us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:40.747776 (+122838us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":184279296}
I20240426 13:20:40.850803 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.787545 24953 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:40.848399 25026 rpcz_store.cc:276] 0426 13:20:36.814658 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.402617 (+587959us) service_pool.cc:225] Handling call
0426 13:20:37.446795 (+ 44178us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:37.986541 (+539746us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.642380 (+2655839us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54001e4954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:40.782093 (+139713us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239773056}
I20240426 13:20:40.855360 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:40.886248 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.855957 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.841383 25619 rpcz_store.cc:276] 0426 13:20:37.150805 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.150991 (+   186us) service_pool.cc:225] Handling call
0426 13:20:37.164696 (+ 13705us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:37.210440 (+ 45744us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:37.254997 (+ 44557us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.326557 (+ 71560us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:37.326633 (+    76us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:37.326709 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:39.764841 (+2438132us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b18001be2c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:20:40.746388 (+981547us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:40.746656 (+   268us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.746832 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":233990,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":3874,"raft.queue_time_us":378,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":10601,"spinlock_wait_cycles":208652544,"thread_start_us":114895,"threads_started":2,"wal-append.queue_time_us":1716130,"wal-append.run_cpu_time_us":1366,"wal-append.run_wall_time_us":1145909}
I20240426 13:20:40.860411 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.809087 28731 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:40.820804 28745 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:40.871356 25029 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20240426 13:20:40.877876 24947 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:40.880240 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.893229 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:40.893802 25620 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20240426 13:20:40.881338 25023 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.886840 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.894832 24952 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:40.895617 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.895617 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.889477 25031 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.889367 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:40.898661 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.895855 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.865589 28998 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:40.888654 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:40.896843 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.904937 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:40.824630 28999 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.899775 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:40.907670 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.908967 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.891019 24953 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:40.909392 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.891922 29016 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:40.821132 29140 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.916880 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.917618 25026 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.918788 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.875630 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:40.913172 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.919644 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:40.895846 28731 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:40.913997 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.914958 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.922799 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.920107 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.924090 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.924885 24953 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:40.915849 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.916273 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:40.925832 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.916682 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:40.913650 24949 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:40.919714 24954 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:40.896184 28937 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:40.921526 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.927798 25024 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.928215 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.926420 24953 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:40.926380 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.929265 25029 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:40.876690 28940 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:40.867952 28997 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.918462 28822 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:40.929708 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:40.948932 24953 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:40.950330 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:40.950922 25623 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:40.930748 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:40.953668 29006 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:40.928335 25620 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:40.933176 24951 raft_consensus.cc:2354] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
I20240426 13:20:40.933172 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:40.933749 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:40.929548 24952 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:40.935238 24947 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:40.903306 29066 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:40.979980 24953 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:40.931088 29117 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:40.927793 28998 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:40.972652 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.992630 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.994856 24947 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:40.993191 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 738) took 3775 ms. Trace:
W20240426 13:20:40.945485 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 379722us
I20240426 13:20:40.931324 28937 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:40.974542 25101 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [9.8-10.9]
I20240426 13:20:40.883304 29176 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.920812 28707 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:40.973114 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.975613 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.977833 25096 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
W20240426 13:20:40.990412 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56738.6us
I20240426 13:20:40.975875 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:40.920287 29016 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:40.962990 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1715) took 2494 ms (client timeout 3299 ms). Trace:
I20240426 13:20:40.955451 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:41.003871 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:40.993499 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
W20240426 13:20:41.039392 25104 rpcz_store.cc:269] 0426 13:20:38.455600 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.608883 (+153283us) service_pool.cc:225] Handling call
0426 13:20:40.949742 (+2340859us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:40.949844 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":629740928}
I20240426 13:20:41.039610 24950 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:41.022508 25101 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:41.027827 25026 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:40.996361 25616 rpcz_store.cc:276] 0426 13:20:37.218065 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.218285 (+   220us) service_pool.cc:225] Handling call
0426 13:20:37.242686 (+ 24401us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:37.256380 (+ 13694us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:37.256427 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:37.341341 (+ 84914us) log.cc:842] Serialized 52 byte log entry
0426 13:20:37.448015 (+106674us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:37.448441 (+   426us) log.cc:842] Serialized 14 byte log entry
0426 13:20:37.448631 (+   190us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:37.448709 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:40.385407 (+2936698us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:40.992911 (+607504us) raft_consensus.cc:1671] finished
0426 13:20:40.993009 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:40.993140 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":102582,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":10196,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":22045,"spinlock_wait_cycles":13143808,"thread_start_us":27438,"threads_started":1,"wal-append.queue_time_us":831221}
I20240426 13:20:40.991854 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:41.022869 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.896292 29090 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.019825 24954 raft_consensus.cc:2396] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:40.992169 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:41.042930 24952 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:40.974934 29006 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:41.045238 25096 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240426 13:20:41.045449 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:40.937117 29165 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:41.003826 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.047605 25623 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:40.990897 28900 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:41.049160 25101 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:41.049465 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20240426 13:20:41.075284 29050 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:40.994004 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:40.993717 29066 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.113766 29064 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:41.113637 29066 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.090587 29103 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:41.115139 29103 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.115468 29103 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:41.051158 29095 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.516s	user 0.002s	sys 0.000s
I20240426 13:20:41.119618 25620 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:41.120185 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.120666 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:41.044361 29009 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.609s	user 0.007s	sys 0.003s
W20240426 13:20:41.053506 29150 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.253s	user 0.002s	sys 0.000s
I20240426 13:20:41.057940 28997 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.058128 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:41.053076 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.061029 24954 pending_rounds.cc:76] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 4
I20240426 13:20:41.061776 24953 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20240426 13:20:41.062646 28999 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:41.063380 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 685) took 1297 ms (client timeout 1539 ms). Trace:
I20240426 13:20:41.068969 29140 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:41.070071 29145 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:41.070240 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2335) took 3790 ms (client timeout 3819 ms). Trace:
I20240426 13:20:41.070675 29035 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:41.082000 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78193.6us
W20240426 13:20:41.082410 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2339) took 2631 ms (client timeout 1634 ms). Trace:
I20240426 13:20:41.082908 24952 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:41.086009 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82242.9us
I20240426 13:20:41.004542 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:40.992660 28940 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:41.050807 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.102733 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:41.149152 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2332) took 3949 ms. Trace:
I20240426 13:20:41.178906 24953 rpcz_store.cc:276] 0426 13:20:37.199678 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.059892 (+3860214us) service_pool.cc:225] Handling call
0426 13:20:41.061527 (+  1635us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:41.148866 (+ 87339us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.149096 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.179592 25027 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.179998 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:41.148391 24954 pending_rounds.cc:97] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20240426 13:20:41.181229 24954 raft_consensus.cc:2866] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:41.181478 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 176990us
I20240426 13:20:41.148504 25616 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:41.154398 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:40.992736 28822 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
W20240426 13:20:41.150656 24956 rpcz_store.cc:269] 0426 13:20:37.279546 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.762946 (+483400us) service_pool.cc:225] Handling call
0426 13:20:41.070058 (+3307112us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.070183 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":933293952}
I20240426 13:20:41.185640 25104 pending_rounds.cc:76] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 7
I20240426 13:20:41.186278 25104 pending_rounds.cc:97] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20240426 13:20:41.186570 25104 raft_consensus.cc:2866] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:41.186681 24956 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [9.8-11.9]
W20240426 13:20:41.182245 25103 rpcz_store.cc:269] 0426 13:20:39.765773 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.766220 (+   447us) service_pool.cc:225] Handling call
0426 13:20:41.063160 (+1296940us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b54003ee9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.063275 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128941312}
I20240426 13:20:41.187351 25623 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:41.187556 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2338) took 2737 ms. Trace:
I20240426 13:20:41.188156 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:41.188835 24956 rpcz_store.cc:276] 0426 13:20:38.450308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.185986 (+2735678us) service_pool.cc:225] Handling call
0426 13:20:41.186389 (+   403us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:41.187238 (+   849us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.187495 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.190508 24956 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:41.191051 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2342) took 2739 ms. Trace:
I20240426 13:20:41.191383 24956 rpcz_store.cc:276] 0426 13:20:38.451315 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.189805 (+2738490us) service_pool.cc:225] Handling call
0426 13:20:41.190136 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.190820 (+   684us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.190999 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.002362 28937 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:41.028827 29176 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.152828 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:41.090418 28660 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:41.145650 24947 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:41.177716 24949 rpcz_store.cc:269] 0426 13:20:38.450570 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.797281 (+1346711us) service_pool.cc:225] Handling call
0426 13:20:41.082239 (+1284958us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.082354 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2391473536}
I20240426 13:20:41.198738 24949 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:41.182487 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:41.103185 28626 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.000s
I20240426 13:20:41.184340 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.198197 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2333) took 3975 ms. Trace:
I20240426 13:20:41.199383 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 709) took 1323 ms. Trace:
I20240426 13:20:41.201032 24947 rpcz_store.cc:276] 0426 13:20:37.222559 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.079878 (+3857319us) service_pool.cc:225] Handling call
0426 13:20:41.145249 (+ 65371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.197974 (+ 52725us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.198152 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.201141 24949 rpcz_store.cc:276] 0426 13:20:39.875941 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.198195 (+1322254us) service_pool.cc:225] Handling call
0426 13:20:41.198454 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.199114 (+   660us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.199319 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.185142 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:41.106564 29111 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.627s	user 0.002s	sys 0.006s
I20240426 13:20:41.097550 29016 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:41.207017 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:41.177551 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2334) took 3929 ms. Trace:
I20240426 13:20:41.193715 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:41.104225 29083 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:41.203055 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.215121 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:41.149907 29008 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:41.193591 29166 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:20:41.197783 28660 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.200389 28626 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.215215 29035 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20240426 13:20:41.220089 29165 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:41.208000 24952 rpcz_store.cc:276] 0426 13:20:37.248666 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.082367 (+3833701us) service_pool.cc:225] Handling call
0426 13:20:41.082712 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.177280 (+ 94568us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.177497 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.223953 29008 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.224505 28626 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.219965 28997 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:41.225441 28997 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.226043 29008 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:41.223122 28660 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:41.226307 28997 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.226817 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.227659 25024 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.228312 25022 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.228510 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.229678 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.208060 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.230406 25024 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.230727 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.231195 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.231472 29035 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:41.232021 29035 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.232239 29083 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.232825 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.232959 29035 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.086071 29045 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:41.234215 28660 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.234652 29176 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:41.235183 29176 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.209978 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.235592 29176 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.149586 29063 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.236505 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.237368 25030 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.238185 28660 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.238384 29063 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.098635 29006 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.238965 29140 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:41.239468 29140 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.239809 29140 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:41.214985 29120 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.240547 29120 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.220615 29111 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.231822 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:41.242436 29120 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.242749 29176 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.243106 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.210415 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:41.243788 25024 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:41.218405 24947 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:41.244486 29165 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:41.183845 29081 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:41.245040 29081 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.245379 29081 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.245761 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.246515 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.247138 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.248250 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.222992 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.250077 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.250619 29176 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.251135 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.233549 25031 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.252820 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.253783 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:41.244688 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 712) took 1274 ms. Trace:
I20240426 13:20:41.254333 24947 rpcz_store.cc:276] 0426 13:20:39.969947 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.217807 (+1247860us) service_pool.cc:225] Handling call
0426 13:20:41.218061 (+   254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.244444 (+ 26383us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.244635 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.254878 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.255475 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:41.153007 29009 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:41.217389 24953 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:41.256405 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.256955 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2337) took 2807 ms. Trace:
I20240426 13:20:41.257508 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.257398 24953 rpcz_store.cc:276] 0426 13:20:38.450046 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.180859 (+2730813us) service_pool.cc:225] Handling call
0426 13:20:41.206077 (+ 25218us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.256651 (+ 50574us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.256891 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.258728 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.214851 28680 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:41.259518 28680 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.259337 25098 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:41.217151 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.248930 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.260344 25101 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:20:41.235284 29045 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:41.220736 24949 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:41.260565 25098 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:41.261324 29176 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:41.262653 29176 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.263141 29176 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:41.263242 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 713) took 1293 ms. Trace:
I20240426 13:20:41.263664 24949 rpcz_store.cc:276] 0426 13:20:39.970183 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.215058 (+1244875us) service_pool.cc:225] Handling call
0426 13:20:41.215436 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.263022 (+ 47586us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.263191 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.197212 29109 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:41.264415 29109 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:41.264812 29109 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:41.262696 29140 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.266695 29122 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:41.239005 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.268205 29096 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:41.266692 29140 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.269174 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.269757 29140 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:41.269934 25023 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.270179 29140 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:41.270850 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.272293 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.272902 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:41.271906 29140 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.273669 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.203595 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:41.274705 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.276032 25031 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.276332 25619 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:41.261116 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:41.276783 24949 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20240426 13:20:41.279100 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:41.279683 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.279361 24949 pending_rounds.cc:76] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:20:41.280872 24949 pending_rounds.cc:97] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240426 13:20:41.281227 24949 raft_consensus.cc:2866] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:41.264047 29081 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.287550 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.285005 25024 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:41.204138 29013 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:41.316793 25616 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 13:20:41.317567 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 745) took 1004 ms. Trace:
I20240426 13:20:41.317981 25616 rpcz_store.cc:276] 0426 13:20:40.312809 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.281740 (+968931us) service_pool.cc:225] Handling call
0426 13:20:41.307986 (+ 26246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.317251 (+  9265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.317505 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:40.970027 28927 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.283790 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.271983 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.287894 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.293805 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.304666 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.234609 25028 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.328186 24949 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:41.328356 25023 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.311090 24950 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
W20240426 13:20:41.312472 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 108985us
I20240426 13:20:41.330490 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:41.200528 28822 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.324436 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.337095 25024 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:41.259912 28680 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.332114 24949 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:41.333340 24950 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:41.335323 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.339443 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:41.341550 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62108us
I20240426 13:20:41.344524 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.316388 29013 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:41.198558 28940 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.122s
I20240426 13:20:41.234230 29083 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:41.343413 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.311676 28626 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:41.350411 25098 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:41.314513 29009 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:41.352806 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:41.385452 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80850.1us
I20240426 13:20:41.385665 25024 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:41.386025 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.352818 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:20:41.384233 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.351436 29013 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:41.388826 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.389549 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.350028 29008 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:41.345403 29110 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20240426 13:20:41.345885 28917 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:41.394657 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.401002 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.401278 29051 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.406421 29051 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.408864 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.409125 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.337186 29066 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.437776 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:41.376649 29187 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:41.435530 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.391000 29045 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:41.441355 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.436911 28626 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:41.444831 28707 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:41.440076 29064 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 13:20:41.451517 29016 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:41.434387 29102 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.023s	sys 0.059s
I20240426 13:20:41.399124 29008 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.447521 25023 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.448868 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:41.450865 28707 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.468819 28707 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:41.440245 28660 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:41.471712 28660 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.472579 28660 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.475183 28707 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.409838 29120 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:41.408607 28917 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:41.476210 29120 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.477172 29120 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:41.477617 28707 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:41.476387 28917 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:41.436025 29063 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:41.468303 29008 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:41.445459 29045 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:41.476050 29013 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:41.456501 29102 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.023s	sys 0.059s
I20240426 13:20:41.456874 29050 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.482107 29187 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:41.486210 29063 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:41.489696 29023 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.012s	sys 0.015s
I20240426 13:20:41.489156 29051 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.493664 29045 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:41.489657 29148 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.011s	sys 0.016s
I20240426 13:20:41.552996 25101 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:41.405606 28900 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.562546 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 534) took 3537 ms. Trace:
W20240426 13:20:41.529593 29023 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.012s	sys 0.016s
I20240426 13:20:41.522917 29058 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:41.528306 29063 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:41.566744 25025 rpcz_store.cc:276] 0426 13:20:38.025515 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.025687 (+   172us) service_pool.cc:225] Handling call
0426 13:20:38.070926 (+ 45239us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.633005 (+1562079us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.557257 (+1924252us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:41.562474 (+  5217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3227380480}
I20240426 13:20:41.479131 28937 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.457479 28822 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.553277 29165 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:41.556592 29187 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:41.617242 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.009s	sys 0.049s
W20240426 13:20:41.617852 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.009s	sys 0.049s
I20240426 13:20:41.617199 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.618678 25025 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20240426 13:20:41.563377 28731 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.629s	user 0.008s	sys 0.002s
I20240426 13:20:41.643319 25024 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:41.643705 29013 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:41.643850 28707 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:41.644785 28707 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:41.645092 28707 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.645656 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.645957 28707 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:41.646858 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:41.647022 28707 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:41.643543 29050 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:41.649446 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2330) took 4497 ms. Trace:
I20240426 13:20:41.647941 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:41.649834 24948 rpcz_store.cc:276] 0426 13:20:37.152383 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.166646 (+1014263us) service_pool.cc:225] Handling call
0426 13:20:38.166985 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.298721 (+1131736us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.649200 (+2350479us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400102054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:41.649391 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2395413376}
I20240426 13:20:41.650409 28937 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.650844 25025 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:41.651274 24948 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:41.651553 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1743) took 1182 ms. Trace:
I20240426 13:20:41.637300 29186 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240426 13:20:41.605070 29081 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:41.638239 28626 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:41.617523 29110 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20240426 13:20:41.624092 29165 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.651876 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1909) took 1204 ms. Trace:
I20240426 13:20:41.677186 24948 rpcz_store.cc:276] 0426 13:20:40.447581 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.650722 (+1203141us) service_pool.cc:225] Handling call
0426 13:20:41.651031 (+   309us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.651657 (+   626us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.651829 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.658084 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2354) took 1230 ms. Trace:
I20240426 13:20:41.678514 24949 rpcz_store.cc:276] 0426 13:20:40.427435 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.329207 (+901772us) service_pool.cc:225] Handling call
0426 13:20:41.331873 (+  2666us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.345611 (+ 13738us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.657865 (+312254us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b54002cbfd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.658005 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5336832}
I20240426 13:20:41.680260 24949 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240426 13:20:41.659063 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 746) took 1346 ms. Trace:
I20240426 13:20:41.682324 25618 rpcz_store.cc:276] 0426 13:20:40.313076 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.441995 (+128919us) service_pool.cc:225] Handling call
0426 13:20:41.658868 (+1216873us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.658975 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1867872640}
I20240426 13:20:41.661641 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 764) took 3599 ms. Trace:
I20240426 13:20:41.683913 25624 rpcz_store.cc:276] 0426 13:20:38.062195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.062371 (+   176us) service_pool.cc:225] Handling call
0426 13:20:38.062757 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:39.946321 (+1883564us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.661440 (+1715119us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:41.661593 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4219993344}
W20240426 13:20:41.665385 25097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2057) took 4669 ms (client timeout 2214 ms). Trace:
W20240426 13:20:41.685432 25097 rpcz_store.cc:269] 0426 13:20:36.996235 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:37.001017 (+  4782us) service_pool.cc:225] Handling call
0426 13:20:41.665189 (+4664172us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54003e17d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.665303 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5893927296}
I20240426 13:20:41.686589 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:41.665979 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1718) took 1253 ms (client timeout 1510 ms). Trace:
W20240426 13:20:41.690220 25102 rpcz_store.cc:269] 0426 13:20:40.412861 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.428778 (+ 15917us) service_pool.cc:225] Handling call
0426 13:20:41.665845 (+1237067us) spinlock_profiling.cc:243] Waited 98 ms on lock 0x7b54003b06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.665932 (+    87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225391872}
I20240426 13:20:41.691390 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:41.669296 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:41.669642 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 705) took 3373 ms (client timeout 1874 ms). Trace:
W20240426 13:20:41.694183 24955 rpcz_store.cc:269] 0426 13:20:38.296274 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.954809 (+1658535us) service_pool.cc:225] Handling call
0426 13:20:41.669493 (+1714684us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.669588 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":332168576}
I20240426 13:20:41.670120 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 742) took 3228 ms. Trace:
I20240426 13:20:41.695561 25621 rpcz_store.cc:276] 0426 13:20:38.441793 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.769050 (+1327257us) service_pool.cc:225] Handling call
0426 13:20:39.780316 (+ 11266us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.964903 (+184587us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.669888 (+1704985us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:41.670066 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274550528}
I20240426 13:20:41.674892 25026 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:41.697880 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1745) took 1207 ms. Trace:
I20240426 13:20:41.698264 25026 rpcz_store.cc:276] 0426 13:20:40.490998 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.674396 (+1183398us) service_pool.cc:225] Handling call
0426 13:20:41.674708 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.697598 (+ 22890us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.697818 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.675534 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1737) took 1251 ms. Trace:
I20240426 13:20:41.651983 25025 rpcz_store.cc:276] 0426 13:20:40.469572 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.650443 (+1180871us) service_pool.cc:225] Handling call
0426 13:20:41.650703 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.651297 (+   594us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.651496 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.699700 25027 rpcz_store.cc:276] 0426 13:20:40.424341 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.424440 (+    99us) service_pool.cc:225] Handling call
0426 13:20:41.675349 (+1250909us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54004173d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:41.675477 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1683049856}
I20240426 13:20:41.687911 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.703667 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1911) took 1255 ms. Trace:
I20240426 13:20:41.704206 24949 rpcz_store.cc:276] 0426 13:20:40.448253 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.679559 (+1231306us) service_pool.cc:225] Handling call
0426 13:20:41.679972 (+   413us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.703399 (+ 23427us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.703610 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.529003 29109 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.729770 24949 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:41.760017 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:41.653944 29111 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:41.757956 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:41.772255 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:41.772897 25624 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.764845 25098 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20240426 13:20:41.768824 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:41.667151 29169 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.176s	user 0.002s	sys 0.007s
I20240426 13:20:41.775126 25026 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:41.777498 24948 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:41.804636 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:41.655716 29066 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.654361 29017 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:41.808892 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:41.809262 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 741) took 3368 ms. Trace:
I20240426 13:20:41.809470 25027 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20240426 13:20:41.809664 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:41.811071 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1750) took 1270 ms. Trace:
I20240426 13:20:41.912883 25025 rpcz_store.cc:276] 0426 13:20:40.540290 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.757497 (+1217207us) service_pool.cc:225] Handling call
0426 13:20:41.810505 (+ 53008us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.810706 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.810742 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:41.810796 (+    54us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:41.810838 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.810902 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:41.811011 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.904945 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1748) took 1392 ms. Trace:
W20240426 13:20:41.909801 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101898us
I20240426 13:20:41.915375 25026 rpcz_store.cc:276] 0426 13:20:40.512505 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.699138 (+1186633us) service_pool.cc:225] Handling call
0426 13:20:41.774949 (+ 75811us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.805819 (+ 30870us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.904852 (+ 99033us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:41.903213 24947 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.006s	sys 0.015s
W20240426 13:20:41.767006 28999 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.012s	sys 0.005s
I20240426 13:20:41.913233 25622 rpcz_store.cc:276] 0426 13:20:38.441065 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.441214 (+   149us) service_pool.cc:225] Handling call
0426 13:20:38.472615 (+ 31401us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:39.349108 (+876493us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:39.375437 (+ 26329us) log.cc:842] Serialized 14 byte log entry
0426 13:20:39.689980 (+314543us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:39.690057 (+    77us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:39.690106 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.809073 (+2118967us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:41.809120 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:41.809215 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":199284,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":82,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":19520,"spinlock_wait_cycles":2015130368,"thread_start_us":314211,"threads_started":1,"wal-append.queue_time_us":410308,"wal-append.run_cpu_time_us":1717,"wal-append.run_wall_time_us":1200842}
W20240426 13:20:41.918869 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.006s	sys 0.015s
I20240426 13:20:41.666173 29030 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:41.706651 29148 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.214s
I20240426 13:20:41.664508 29124 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:41.937388 28429 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:41.937841 28429 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:41.938834 28429 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:41.940613 29188 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.285s	user 0.000s	sys 0.000s
I20240426 13:20:41.934895 25622 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:41.943418 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 545) took 1470 ms. Trace:
I20240426 13:20:41.943771 25622 rpcz_store.cc:276] 0426 13:20:40.473115 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.934258 (+1461143us) service_pool.cc:225] Handling call
0426 13:20:41.934565 (+   307us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.943115 (+  8550us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.943354 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.925371 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:41.926337 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1751) took 1378 ms. Trace:
I20240426 13:20:41.952126 25027 rpcz_store.cc:276] 0426 13:20:40.547884 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.758398 (+1210514us) service_pool.cc:225] Handling call
0426 13:20:41.809265 (+ 50867us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.925997 (+116732us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.926273 (+   276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:41.664386 29165 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:41.768647 28998 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:41.774900 29090 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:41.933482 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.018s	sys 0.013s
W20240426 13:20:41.999199 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.018s	sys 0.014s
W20240426 13:20:41.999487 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 669042us
I20240426 13:20:41.933684 29064 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 13:20:41.921324 28999 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.012s	sys 0.006s
I20240426 13:20:41.904105 29050 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.125s
I20240426 13:20:41.935675 25025 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:41.931418 29030 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:41.936017 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2078) took 2053 ms. Trace:
I20240426 13:20:41.676538 29009 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.267s
I20240426 13:20:41.641778 28731 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.931911 29017 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:41.759512 29187 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:42.028990 28731 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:42.029287 29017 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:42.030413 29187 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:41.985589 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.010s	sys 0.022s
I20240426 13:20:41.701833 29186 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.221s
I20240426 13:20:41.990947 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.032243 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.032361 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:41.992086 24949 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:42.033417 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1738) took 1205 ms. Trace:
I20240426 13:20:42.033409 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.035115 29006 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:42.033823 25105 rpcz_store.cc:276] 0426 13:20:40.827653 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.896180 (+ 68527us) service_pool.cc:225] Handling call
0426 13:20:42.032211 (+1136031us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:42.033089 (+   878us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3425664}
I20240426 13:20:42.035569 29006 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:41.994448 25098 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:42.036348 25098 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:42.036481 29006 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.034049 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.038367 24949 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.999430 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:42.040336 25621 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:42.041254 29109 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:41.903898 28997 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.003s	sys 0.019s
W20240426 13:20:42.041939 28997 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.003s	sys 0.019s
I20240426 13:20:42.024681 25099 rpcz_store.cc:276] 0426 13:20:39.882802 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:39.896515 (+ 13713us) service_pool.cc:225] Handling call
0426 13:20:39.910428 (+ 13913us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:39.928304 (+ 17876us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:39.928335 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:39.928705 (+   370us) log.cc:842] Serialized 26 byte log entry
0426 13:20:40.335931 (+407226us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:40.335987 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:40.336096 (+   109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:41.666661 (+1330565us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:41.935736 (+269075us) raft_consensus.cc:1671] finished
0426 13:20:41.935757 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:41.935962 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17448,"raft.queue_time_us":18845,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":23065856,"thread_start_us":403748,"threads_started":1,"wal-append.queue_time_us":489873}
I20240426 13:20:42.026155 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1766) took 1192 ms. Trace:
I20240426 13:20:42.035152 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:42.000277 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:42.043071 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:42.043637 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.011s	sys 0.022s
I20240426 13:20:42.006896 29037 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:42.037402 25098 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:42.041370 25621 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:41.986788 29066 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20240426 13:20:42.043799 25622 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:41.768872 29134 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.230s
I20240426 13:20:41.996848 29130 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:42.042892 25025 rpcz_store.cc:276] 0426 13:20:40.833688 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.914705 (+1081017us) service_pool.cc:225] Handling call
0426 13:20:41.935399 (+ 20694us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.024559 (+ 89160us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.026092 (+  1533us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:42.044035 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 795496us
I20240426 13:20:41.659027 28626 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:42.087795 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:42.046428 25623 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.002s	sys 0.032s
I20240426 13:20:42.048244 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:42.048753 25098 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:42.091786 25623 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.002s	sys 0.032s
W20240426 13:20:42.046721 24950 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.017s	sys 0.021s
W20240426 13:20:42.092959 24950 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.017s	sys 0.021s
I20240426 13:20:42.052095 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 548) took 1509 ms. Trace:
I20240426 13:20:42.072413 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:42.094548 25621 rpcz_store.cc:276] 0426 13:20:40.542804 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.040915 (+1498111us) service_pool.cc:225] Handling call
0426 13:20:42.041205 (+   290us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.051814 (+ 10609us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.052009 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:42.095279 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 775) took 2193 ms. Trace:
W20240426 13:20:42.046677 29174 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.010s	sys 0.024s
W20240426 13:20:42.108217 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 60065.6us
I20240426 13:20:42.097087 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.048732 28998 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:42.100595 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:42.071738 28906 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:42.080492 28680 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.729s	user 0.004s	sys 0.005s
I20240426 13:20:42.102675 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:42.102890 29016 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:42.080500 28660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.018s	sys 0.020s
I20240426 13:20:42.167793 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.180287 24948 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:42.172503 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69905.9us
I20240426 13:20:42.181437 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:42.110786 29174 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.010s	sys 0.025s
I20240426 13:20:42.096225 29010 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20240426 13:20:42.089344 29124 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20240426 13:20:42.110643 29130 consensus_queue.cc:1225] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:20:42.177901 25621 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:42.202386 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 549) took 1619 ms. Trace:
I20240426 13:20:42.202859 25621 rpcz_store.cc:276] 0426 13:20:40.587567 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.098313 (+1510746us) service_pool.cc:225] Handling call
0426 13:20:42.171657 (+ 73344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.202029 (+ 30372us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.202313 (+   284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:42.104761 29093 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20240426 13:20:42.105816 25623 rpcz_store.cc:276] 0426 13:20:39.902022 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.189683 (+1287661us) service_pool.cc:225] Handling call
0426 13:20:41.217714 (+ 28031us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.221062 (+  3348us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.221103 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.249003 (+ 27900us) log.cc:842] Serialized 26 byte log entry
0426 13:20:42.092723 (+843720us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:42.092773 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.095034 (+  2261us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:42.095100 (+    66us) raft_consensus.cc:1671] finished
0426 13:20:42.095112 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:42.095230 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2940,"raft.queue_time_us":14210,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":843326,"threads_started":1,"wal-append.queue_time_us":778520}
I20240426 13:20:42.090387 29104 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.079s
I20240426 13:20:42.213284 29130 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:42.180987 29094 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:42.216698 29094 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:42.217170 29094 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:42.168293 29197 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:42.079422 29037 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.096158 28870 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.084s
I20240426 13:20:42.048637 29090 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.184671 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:42.168104 29009 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:42.140475 28906 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:42.167764 29111 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:42.205998 25623 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:42.177683 29035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.037s	sys 0.041s
W20240426 13:20:42.185029 28660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.019s	sys 0.020s
W20240426 13:20:42.170414 25103 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.015s	sys 0.045s
W20240426 13:20:42.316740 25103 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.015s	sys 0.045s
W20240426 13:20:42.317215 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.012s	sys 0.025s
W20240426 13:20:42.317615 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.012s	sys 0.025s
I20240426 13:20:42.222494 24949 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:42.318655 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.225912 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1762) took 1393 ms. Trace:
I20240426 13:20:42.319730 25024 rpcz_store.cc:276] 0426 13:20:40.832692 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.386620 (+553928us) service_pool.cc:225] Handling call
0426 13:20:42.225693 (+839073us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54002f9c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:42.225839 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":527920896}
I20240426 13:20:42.226282 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1741) took 1784 ms. Trace:
I20240426 13:20:42.321038 25022 rpcz_store.cc:276] 0426 13:20:40.442185 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.584087 (+141902us) service_pool.cc:225] Handling call
0426 13:20:42.226091 (+1642004us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:42.226220 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":997959424}
W20240426 13:20:42.226806 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 546) took 1753 ms (client timeout 2035 ms). Trace:
I20240426 13:20:42.227483 25620 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:42.228582 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1755) took 1639 ms (client timeout 1901 ms). Trace:
W20240426 13:20:42.322157 25030 rpcz_store.cc:269] 0426 13:20:40.589458 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.786404 (+196946us) service_pool.cc:225] Handling call
0426 13:20:42.228420 (+1442016us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54000b0c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:42.228531 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":844715392}
I20240426 13:20:42.229197 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:42.321970 25619 rpcz_store.cc:269] 0426 13:20:40.473406 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.888356 (+414950us) service_pool.cc:225] Handling call
0426 13:20:42.226605 (+1338249us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:42.226731 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829517568}
I20240426 13:20:42.323448 25098 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:42.324338 25619 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:42.230487 25097 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:42.225544 29197 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:42.245328 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:42.175045 28680 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:42.249744 25102 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240426 13:20:42.251442 29165 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:42.256654 29196 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20240426 13:20:42.316900 29010 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:42.267684 29063 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.661s	user 0.005s	sys 0.001s
I20240426 13:20:42.266888 24948 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:42.364787 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:42.365726 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:42.351894 28707 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.704s	user 0.010s	sys 0.001s
I20240426 13:20:42.350008 29197 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:42.273036 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:42.187786 29081 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:42.142306 29192 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.109s
I20240426 13:20:42.308614 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 779) took 1354 ms. Trace:
W20240426 13:20:42.310940 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 126372us
I20240426 13:20:42.330377 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:42.330905 25098 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:42.331504 25022 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
W20240426 13:20:42.331671 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.022s	sys 0.015s
I20240426 13:20:42.332233 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 780) took 1333 ms. Trace:
W20240426 13:20:42.232702 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1824977979999858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3004s for a response
W20240426 13:20:42.317891 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 705542us
W20240426 13:20:42.304471 29035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.038s	sys 0.042s
I20240426 13:20:42.349190 25097 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20240426 13:20:42.349751 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.006s	sys 0.015s
I20240426 13:20:42.309334 28906 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:42.309116 29111 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:42.257611 29194 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
W20240426 13:20:42.345042 29103 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.229s	user 0.011s	sys 0.001s
I20240426 13:20:42.367080 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.532582 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.533277 24948 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:42.107532 29058 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
W20240426 13:20:42.379945 25624 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.603s	user 0.008s	sys 0.000s
I20240426 13:20:42.276369 25622 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:42.213274 28998 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:42.486652 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.006s	sys 0.016s
I20240426 13:20:42.537736 25621 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
W20240426 13:20:42.537779 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 913810us
W20240426 13:20:42.474397 28917 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.996s	user 0.009s	sys 0.000s
I20240426 13:20:42.485224 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2100) took 1198 ms. Trace:
I20240426 13:20:42.485934 25027 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:42.552488 24949 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
W20240426 13:20:42.362071 28716 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.007s	sys 0.017s
I20240426 13:20:42.461279 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2107) took 1171 ms. Trace:
W20240426 13:20:42.487321 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.054s	sys 0.086s
I20240426 13:20:42.481906 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1752) took 1918 ms. Trace:
I20240426 13:20:42.482259 27911 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:42.489156 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:42.319023 29009 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.475700 29037 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:42.510896 25025 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:42.462314 29045 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.893s	user 0.004s	sys 0.004s
W20240426 13:20:42.462988 29200 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
W20240426 13:20:42.545622 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.022s	sys 0.020s
W20240426 13:20:42.547180 24921 connection.cc:232] RPC call timeout handler was delayed by -1.0789977980000458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31190000000000007s for a response
I20240426 13:20:42.572295 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:42.509356 29205 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:42.460726 29171 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.682s	user 0.008s	sys 0.002s
I20240426 13:20:42.458269 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 772) took 2586 ms. Trace:
W20240426 13:20:42.474326 29120 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.996s	user 0.009s	sys 0.000s
I20240426 13:20:42.458515 29193 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:42.545962 25623 rpcz_store.cc:276] 0426 13:20:40.953734 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.205568 (+1251834us) service_pool.cc:225] Handling call
0426 13:20:42.205840 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.308312 (+102472us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.308548 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:42.592867 25617 rpcz_store.cc:276] 0426 13:20:39.871788 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.161534 (+289746us) service_pool.cc:225] Handling call
0426 13:20:40.171689 (+ 10155us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:40.559739 (+388050us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.457996 (+1898257us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:42.458198 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":859486464}
I20240426 13:20:42.605098 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:42.570969 28716 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.007s	sys 0.017s
I20240426 13:20:42.554384 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:42.571502 27911 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:42.547894 25619 rpcz_store.cc:276] 0426 13:20:40.998777 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.323752 (+1324975us) service_pool.cc:225] Handling call
0426 13:20:42.324116 (+   364us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.331938 (+  7822us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.332169 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:42.554010 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2465) took 1677 ms. Trace:
I20240426 13:20:42.541229 29103 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:42.473309 28680 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:42.552057 25103 rpcz_store.cc:276] 0426 13:20:41.286932 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.290037 (+  3105us) service_pool.cc:225] Handling call
0426 13:20:41.290394 (+   357us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.314310 (+ 23916us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:41.345582 (+ 31272us) log.cc:842] Serialized 14 byte log entry
0426 13:20:42.317231 (+971649us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:42.328827 (+ 11596us) log.cc:842] Serialized 26 byte log entry
0426 13:20:42.329761 (+   934us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:42.329808 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.329880 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:42.484978 (+155098us) raft_consensus.cc:1671] finished
0426 13:20:42.485003 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:42.485161 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33815,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":256,"raft.run_cpu_time_us":418,"raft.run_wall_time_us":41699,"thread_start_us":927474,"threads_started":1,"wal-append.queue_time_us":1113709}
I20240426 13:20:42.573974 28917 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:42.573091 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.063s	sys 0.088s
I20240426 13:20:42.575289 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1737) took 1748 ms. Trace:
I20240426 13:20:42.387773 29081 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:42.536243 25624 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:42.573182 25101 rpcz_store.cc:276] 0426 13:20:41.289436 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.340620 (+ 51184us) service_pool.cc:225] Handling call
0426 13:20:41.340842 (+   222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.567581 (+226739us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.461060 (+893479us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:42.461227 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434566528}
I20240426 13:20:42.470821 28626 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:42.459064 29184 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:42.559264 24948 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:42.690912 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 781) took 1036 ms. Trace:
W20240426 13:20:42.694896 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.016s	sys 0.012s
W20240426 13:20:42.475785 29176 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.212s	user 0.008s	sys 0.004s
I20240426 13:20:42.363425 29165 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.469601 29090 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:42.571805 25028 rpcz_store.cc:276] 0426 13:20:40.563960 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.585445 (+ 21485us) service_pool.cc:225] Handling call
0426 13:20:42.481686 (+1896241us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b5400241154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:42.481842 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":888116480}
W20240426 13:20:42.615999 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.010s	sys 0.022s
W20240426 13:20:42.612641 24953 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.017s	sys 0.037s
W20240426 13:20:42.577253 25104 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.010s	user 0.008s	sys 0.046s
I20240426 13:20:42.618569 25623 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20240426 13:20:42.616137 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.004s	sys 0.029s
W20240426 13:20:42.610805 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56522.9us
I20240426 13:20:42.368570 29063 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.639200 27911 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:42.509948 29111 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:42.635211 25022 rpcz_store.cc:276] 0426 13:20:40.876244 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.330948 (+1454704us) service_pool.cc:225] Handling call
0426 13:20:42.331286 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.546745 (+215459us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.553951 (+  7206us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:42.605683 28999 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.018s	sys 0.000s
I20240426 13:20:42.581130 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.590286 29058 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.120s
I20240426 13:20:42.607360 29205 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.250167 29006 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:43.251017 24953 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.378s	user 0.038s	sys 0.057s
I20240426 13:20:42.704479 29184 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:42.693081 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 46819.8us
W20240426 13:20:43.253397 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.326s	user 0.015s	sys 0.052s
I20240426 13:20:43.000932 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2331) took 5829 ms. Trace:
W20240426 13:20:43.254132 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.519s	user 0.034s	sys 0.066s
W20240426 13:20:43.254711 25104 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.689s	user 0.012s	sys 0.078s
W20240426 13:20:43.025684 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 739) took 4590 ms (client timeout 1624 ms). Trace:
I20240426 13:20:43.026067 25616 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:43.042001 25026 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20240426 13:20:43.041817 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1754) took 2458 ms. Trace:
I20240426 13:20:43.256520 25023 rpcz_store.cc:276] 0426 13:20:40.583303 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.347406 (+764103us) service_pool.cc:225] Handling call
0426 13:20:43.041602 (+1694196us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.041747 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138035456}
I20240426 13:20:42.694777 25624 rpcz_store.cc:276] 0426 13:20:41.654716 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.771961 (+117245us) service_pool.cc:225] Handling call
0426 13:20:42.690828 (+918867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.025108 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 553) took 2156 ms. Trace:
I20240426 13:20:43.258374 25621 rpcz_store.cc:276] 0426 13:20:40.868747 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.204439 (+1335692us) service_pool.cc:225] Handling call
0426 13:20:42.537532 (+333093us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.538259 (+   727us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.024852 (+486593us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54007521d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.025038 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":746039040}
W20240426 13:20:43.040280 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 714) took 2965 ms (client timeout 2172 ms). Trace:
W20240426 13:20:43.257771 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.577s	user 0.035s	sys 0.066s
W20240426 13:20:42.695611 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.016s	sys 0.012s
W20240426 13:20:42.756611 25588 leader_election.cc:334] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.172s (SENT)
W20240426 13:20:43.253940 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.33668e+06us
I20240426 13:20:43.265969 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:42.605984 29192 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:43.315987 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:43.260186 24951 rpcz_store.cc:269] 0426 13:20:40.075350 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.426037 (+350687us) service_pool.cc:225] Handling call
0426 13:20:43.039989 (+2613952us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54003679d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.040175 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993200000}
I20240426 13:20:43.047585 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2463) took 2213 ms. Trace:
I20240426 13:20:43.260813 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 716) took 3111 ms. Trace:
I20240426 13:20:42.691713 25100 rpcz_store.cc:276] 0426 13:20:40.827340 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.063234 (+235894us) service_pool.cc:225] Handling call
0426 13:20:41.063543 (+   309us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.063874 (+   331us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.063903 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.068529 (+  4626us) log.cc:842] Serialized 26 byte log entry
0426 13:20:41.518532 (+450003us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:41.518585 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.518690 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:42.575104 (+1056414us) raft_consensus.cc:1671] finished
0426 13:20:42.575123 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:42.575238 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3331,"raft.queue_time_us":18452,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":64199,"thread_start_us":449612,"threads_started":1,"wal-append.queue_time_us":853574}
W20240426 13:20:43.274062 24956 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.506s	user 0.022s	sys 0.028s
I20240426 13:20:42.590953 29045 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:43.264874 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.323s	user 0.004s	sys 0.037s
I20240426 13:20:43.046068 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2462) took 2211 ms. Trace:
I20240426 13:20:43.254045 24954 rpcz_store.cc:276] 0426 13:20:37.171721 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.058740 (+3887019us) service_pool.cc:225] Handling call
0426 13:20:41.060245 (+  1505us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.181737 (+121492us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.181767 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.182162 (+   395us) log.cc:842] Serialized 26 byte log entry
0426 13:20:41.680431 (+498269us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:41.680466 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:41.680528 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:43.000661 (+1320133us) raft_consensus.cc:1671] finished
0426 13:20:43.000690 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.000851 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":441757,"threads_started":1,"wal-append.queue_time_us":1008456}
I20240426 13:20:43.338932 25027 rpcz_store.cc:276] 0426 13:20:40.834220 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.953135 (+1118915us) service_pool.cc:225] Handling call
0426 13:20:42.485721 (+532586us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.552728 (+ 67007us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.045851 (+493123us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400241150. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.046004 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1061393280}
I20240426 13:20:42.707031 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:43.317849 24956 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.550s	user 0.024s	sys 0.030s
W20240426 13:20:43.280921 24995 connection.cc:232] RPC call timeout handler was delayed by -1.0130336480000324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20240426 13:20:43.317375 24952 rpcz_store.cc:276] 0426 13:20:40.149704 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.225175 (+1075471us) service_pool.cc:225] Handling call
0426 13:20:41.225429 (+   254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.225627 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:41.680978 (+455351us) log.cc:842] Serialized 14 byte log entry
0426 13:20:43.260502 (+1579524us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:43.260551 (+    49us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:43.260599 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.260661 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.260770 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":801758,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":6167,"thread_start_us":2034248,"threads_started":2,"wal-append.queue_time_us":1570081}
I20240426 13:20:43.316790 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 542) took 2881 ms. Trace:
I20240426 13:20:43.341233 25616 rpcz_store.cc:276] 0426 13:20:40.435312 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.337392 (+902080us) service_pool.cc:225] Handling call
0426 13:20:43.025832 (+1688440us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.256226 (+230394us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.316616 (+ 60390us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54007503d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.316738 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3871660288}
I20240426 13:20:43.341516 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:43.342290 24952 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:43.343115 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:43.320475 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 786) took 1285 ms. Trace:
I20240426 13:20:42.637431 29010 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:43.017055 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 777) took 2950 ms. Trace:
I20240426 13:20:43.344233 25623 rpcz_store.cc:276] 0426 13:20:42.034693 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.617756 (+583063us) service_pool.cc:225] Handling call
0426 13:20:42.618125 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.320137 (+702012us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.320413 (+   276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.344560 25620 rpcz_store.cc:276] 0426 13:20:40.066245 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.205907 (+1139662us) service_pool.cc:225] Handling call
0426 13:20:42.227242 (+1021335us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.326507 (+ 99265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.016812 (+690305us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54006be9d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.016984 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2305957248}
W20240426 13:20:42.777364 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.014s	sys 0.048s
W20240426 13:20:43.346539 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.340s	user 0.023s	sys 0.066s
I20240426 13:20:42.858299 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 698) took 1338 ms. Trace:
I20240426 13:20:43.348217 25096 rpcz_store.cc:276] 0426 13:20:41.520389 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.520679 (+   290us) service_pool.cc:225] Handling call
0426 13:20:41.559119 (+ 38440us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.559381 (+   262us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.559417 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.571032 (+ 11615us) log.cc:842] Serialized 26 byte log entry
0426 13:20:42.588872 (+1017840us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:42.588926 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:42.589013 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:42.857997 (+268984us) raft_consensus.cc:1671] finished
0426 13:20:42.858027 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:42.858217 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":975054,"threads_started":1,"wal-append.queue_time_us":1097097}
I20240426 13:20:42.758702 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:20:42.855230 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:42.730423 25029 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:43.255920 25615 rpcz_store.cc:269] 0426 13:20:38.434794 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:38.438512 (+  3718us) service_pool.cc:225] Handling call
0426 13:20:43.025420 (+4586908us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.025589 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4307007744}
I20240426 13:20:42.889134 25098 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 13:20:42.823808 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:43.292021 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2356) took 2857 ms. Trace:
I20240426 13:20:43.319437 25025 rpcz_store.cc:276] 0426 13:20:40.834453 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.073838 (+1239385us) service_pool.cc:225] Handling call
0426 13:20:42.097414 (+ 23576us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.576021 (+478607us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.047299 (+471278us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:43.047494 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":950381312}
I20240426 13:20:43.048629 29114 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:42.997444 29074 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:42.688995 29050 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.025s	sys 0.044s
I20240426 13:20:42.878225 29018 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:43.283676 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:43.335261 25624 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:42.690590 29023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.076s	user 0.039s	sys 0.041s
W20240426 13:20:42.616225 28927 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.013s	sys 0.019s
I20240426 13:20:43.321019 25022 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:42.651957 29008 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.147s	user 0.011s	sys 0.000s
I20240426 13:20:43.351501 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:43.412554 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2468) took 2484 ms. Trace:
I20240426 13:20:43.412739 25616 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:43.413012 25022 rpcz_store.cc:276] 0426 13:20:40.928433 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.320415 (+2391982us) service_pool.cc:225] Handling call
0426 13:20:43.320801 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.412268 (+ 91467us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.412488 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.414285 25027 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:43.358772 24950 rpcz_store.cc:276] 0426 13:20:40.434234 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.339034 (+904800us) service_pool.cc:225] Handling call
0426 13:20:41.339422 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:41.339780 (+   358us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:42.093482 (+753702us) log.cc:842] Serialized 14 byte log entry
0426 13:20:43.291577 (+1198095us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:43.291657 (+    80us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:43.291719 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.291797 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.291957 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30362,"prepare.queue_time_us":687850,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":30489,"thread_start_us":1920331,"threads_started":2}
I20240426 13:20:43.263962 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:42.841415 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:42.734962 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:42.638090 29202 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:42.611752 29120 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.368366 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:43.210460 29197 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.000s
W20240426 13:20:43.058257 29212 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.445s	user 0.002s	sys 0.000s
I20240426 13:20:42.641969 29130 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.762s
I20240426 13:20:43.374627 25028 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:43.375092 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:43.061825 29199 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.717s	user 0.001s	sys 0.001s
W20240426 13:20:43.037966 29094 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.820s	user 0.011s	sys 0.000s
I20240426 13:20:43.250231 28808 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:43.376592 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s	user 0.038s	sys 0.064s
I20240426 13:20:43.403442 25620 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:43.460407 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s	user 0.047s	sys 0.081s
I20240426 13:20:43.337555 29140 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:43.329082 28999 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.303s	user 0.019s	sys 0.013s
I20240426 13:20:43.403044 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:43.467721 24951 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:43.408118 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:43.410645 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:43.469563 24954 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:43.021036 29208 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:43.086522 29148 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.415789 25103 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:43.416079 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2476) took 2383 ms. Trace:
I20240426 13:20:42.610369 29193 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:43.416299 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 152382us
I20240426 13:20:43.416779 25617 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:42.872565 29035 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.368163 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:43.459347 29122 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:43.317169 29081 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:43.316383 29037 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:43.014948 29126 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:43.388944 28906 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.846s	user 0.005s	sys 0.005s
I20240426 13:20:42.878811 29009 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:43.408497 29050 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.335s	user 0.032s	sys 0.086s
W20240426 13:20:43.352172 28822 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.298s	user 0.050s	sys 0.034s
I20240426 13:20:43.376729 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:43.453099 24952 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:43.455077 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 556) took 2112 ms (client timeout 1626 ms). Trace:
W20240426 13:20:43.545493 25618 rpcz_store.cc:269] 0426 13:20:41.342784 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.758395 (+415611us) service_pool.cc:225] Handling call
0426 13:20:43.454805 (+1696410us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.454937 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944671616}
I20240426 13:20:43.408655 29074 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.351177 29043 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:43.459304 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2467) took 2533 ms. Trace:
I20240426 13:20:43.551560 25028 rpcz_store.cc:276] 0426 13:20:40.925528 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.315543 (+2390015us) service_pool.cc:225] Handling call
0426 13:20:43.374452 (+ 58909us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.458994 (+ 84542us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.459230 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:43.411473 29023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.798s	user 0.048s	sys 0.065s
I20240426 13:20:43.376165 28429 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:43.404413 25023 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20240426 13:20:43.411620 28927 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.430s	user 0.027s	sys 0.046s
I20240426 13:20:43.367604 29114 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:43.367277 29192 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:43.466804 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:43.460078 28900 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.075s	user 0.003s	sys 0.014s
I20240426 13:20:43.371268 29103 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.570798 29192 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.575606 25615 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20240426 13:20:43.550034 29050 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.317039 29176 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.482167 28753 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:43.449676 29018 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:43.483382 25027 rpcz_store.cc:276] 0426 13:20:41.032583 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.340150 (+2307567us) service_pool.cc:225] Handling call
0426 13:20:43.414071 (+ 73921us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.415487 (+  1416us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.415994 (+   507us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:42.690505 29204 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.846s
I20240426 13:20:42.819350 29110 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.530925 29035 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:43.531575 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.334s	user 0.008s	sys 0.046s
I20240426 13:20:43.532615 29165 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.022379 29030 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.531616 29063 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:43.533253 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 697) took 2039 ms. Trace:
W20240426 13:20:43.537221 25105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.062s	user 0.015s	sys 0.016s
I20240426 13:20:43.417683 29008 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.476181 29208 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:43.544709 24948 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:43.483881 29184 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.554144 27911 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:43.544260 28822 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.507s	user 0.050s	sys 0.034s
W20240426 13:20:43.555007 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2469) took 2581 ms (client timeout 1696 ms). Trace:
I20240426 13:20:43.465616 29094 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.560740 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2474) took 2534 ms. Trace:
I20240426 13:20:43.564886 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2461) took 2731 ms. Trace:
I20240426 13:20:43.460435 29197 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:43.567075 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.925s	user 0.017s	sys 0.059s
I20240426 13:20:43.556988 29043 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.536253 29148 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.541064 28906 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.543787 29037 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.556918 29193 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.404604 29010 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:43.567638 28900 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.187s	user 0.003s	sys 0.014s
I20240426 13:20:43.795639 25023 rpcz_store.cc:276] 0426 13:20:41.026594 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.329871 (+2303277us) service_pool.cc:225] Handling call
0426 13:20:43.404106 (+ 74235us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.560405 (+156299us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.560674 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:43.794886 25029 rpcz_store.cc:269] 0426 13:20:40.975257 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.052817 (+ 77560us) service_pool.cc:225] Handling call
0426 13:20:43.554824 (+2502007us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400241654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.554944 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3636874496}
W20240426 13:20:43.796250 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.021s	sys 0.063s
W20240426 13:20:43.795754 25105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.321s	user 0.016s	sys 0.027s
I20240426 13:20:43.716835 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:43.665505 25031 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:43.799264 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.123s	user 0.004s	sys 0.037s
W20240426 13:20:43.585847 25619 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.012s	sys 0.015s
W20240426 13:20:43.574951 29083 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.188s	user 0.008s	sys 0.043s
W20240426 13:20:43.744612 29231 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20240426 13:20:43.794427 25026 rpcz_store.cc:276] 0426 13:20:40.833969 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.916558 (+1082589us) service_pool.cc:225] Handling call
0426 13:20:43.041747 (+1125189us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.269091 (+227344us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.564728 (+295637us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003af7d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:43.564839 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2541791488}
I20240426 13:20:43.781337 24954 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:43.827520 24951 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:43.699479 29220 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.822461 25621 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20240426 13:20:43.806794 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.131s	user 0.004s	sys 0.037s
W20240426 13:20:43.851857 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.17623e+06us
I20240426 13:20:43.779266 29045 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:43.807053 25619 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.079s	user 0.012s	sys 0.021s
W20240426 13:20:43.814451 25102 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.266s	user 0.004s	sys 0.016s
W20240426 13:20:43.804971 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88264us
W20240426 13:20:43.864773 25102 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.316s	user 0.004s	sys 0.016s
I20240426 13:20:43.865147 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:43.866967 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:43.794550 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.810s	user 0.008s	sys 0.055s
W20240426 13:20:43.824905 29083 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.439s	user 0.008s	sys 0.052s
I20240426 13:20:43.848343 25026 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:43.848990 24952 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:20:43.796396 25104 rpcz_store.cc:276] 0426 13:20:41.493463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.493631 (+   168us) service_pool.cc:225] Handling call
0426 13:20:41.525630 (+ 31999us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.526041 (+   411us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.526079 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.526541 (+   462us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.255629 (+1729088us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:43.255685 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.255777 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:43.533015 (+277238us) raft_consensus.cc:1671] finished
0426 13:20:43.533040 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.533191 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1186,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":27605,"thread_start_us":1689660,"threads_started":1,"wal-append.queue_time_us":1845209}
I20240426 13:20:43.849478 25099 raft_consensus.cc:2354] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:43.833494 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2370) took 1161 ms. Trace:
I20240426 13:20:43.850370 25097 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 9. Ops: []
I20240426 13:20:43.699301 29218 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.853540 25098 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:43.814512 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.225s	user 0.010s	sys 0.010s
I20240426 13:20:43.854305 25029 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:43.702724 29230 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.874869 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:43.913079 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 547) took 2698 ms. Trace:
I20240426 13:20:43.913254 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:43.875458 27911 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:43.914407 27911 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:43.914748 27911 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:43.825608 29120 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:43.453480 29202 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:43.882902 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.293s	user 0.010s	sys 0.010s
I20240426 13:20:43.883006 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2498) took 2542 ms. Trace:
I20240426 13:20:43.827945 29058 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:43.847182 29221 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20240426 13:20:43.823309 28822 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.910449 29218 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.913518 25029 rpcz_store.cc:276] 0426 13:20:41.214284 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.802207 (+2587923us) service_pool.cc:225] Handling call
0426 13:20:43.854142 (+ 51935us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.912833 (+ 58691us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.913017 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.915035 27911 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:43.879397 24954 rpcz_store.cc:276] 0426 13:20:42.672131 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.339392 (+667261us) service_pool.cc:225] Handling call
0426 13:20:43.833412 (+494020us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:43.861219 29089 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.865s	user 0.005s	sys 0.006s
I20240426 13:20:43.853672 29193 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:43.920545 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.33108e+06us
I20240426 13:20:43.920565 25026 rpcz_store.cc:276] 0426 13:20:41.340746 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.826192 (+2485446us) service_pool.cc:225] Handling call
0426 13:20:43.848168 (+ 21976us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.882752 (+ 34584us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.882955 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.588845 29050 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.937752 27911 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.925479 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:43.940750 25029 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:43.941407 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2345) took 4138 ms. Trace:
I20240426 13:20:43.801198 29110 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.941948 24956 rpcz_store.cc:276] 0426 13:20:39.802803 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.194030 (+1391227us) service_pool.cc:225] Handling call
0426 13:20:41.206214 (+ 12184us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.622998 (+416784us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.623033 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.640315 (+ 17282us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.340707 (+1700392us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:43.340742 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.340802 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:43.941209 (+600407us) raft_consensus.cc:1671] finished
0426 13:20:43.941230 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.941352 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":60858,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1972858,"threads_started":2,"wal-append.queue_time_us":1868304}
I20240426 13:20:43.916776 28745 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
W20240426 13:20:43.940940 29089 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.006s	sys 0.006s
I20240426 13:20:43.962455 29179 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:43.964160 25617 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
W20240426 13:20:43.964831 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 755) took 1296 ms (client timeout 1509 ms). Trace:
I20240426 13:20:43.965116 29037 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:43.965294 25617 rpcz_store.cc:269] 0426 13:20:42.668017 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.699289 (+ 31272us) service_pool.cc:225] Handling call
0426 13:20:43.964779 (+1265490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:43.862803 29220 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.973490 29050 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.974301 29220 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:43.975529 29223 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20240426 13:20:43.977341 28753 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:43.978140 29008 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:43.979609 29043 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.985960 24956 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:43.986594 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:43.824304 29010 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.945345 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.001825 25026 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:44.002890 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.003751 25026 raft_consensus.cc:2354] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:43.742280 29114 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:43.919896 29204 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.824936 29081 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.007473 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2361) took 3521 ms. Trace:
W20240426 13:20:43.940248 28626 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.010s
I20240426 13:20:43.951406 29005 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.009645 24955 rpcz_store.cc:276] 0426 13:20:40.486165 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.695057 (+1208892us) service_pool.cc:225] Handling call
0426 13:20:41.767478 (+ 72421us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.769153 (+  1675us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.769183 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.812308 (+ 43125us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.347052 (+1534744us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:43.347103 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.347187 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:44.007233 (+660046us) raft_consensus.cc:1671] finished
0426 13:20:44.007260 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.007421 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21491,"raft.queue_time_us":7284,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":101619,"thread_start_us":1340270,"threads_started":1,"wal-append.queue_time_us":1741218}
I20240426 13:20:43.952733 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2346) took 3581 ms. Trace:
W20240426 13:20:44.011245 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.012s	sys 0.042s
I20240426 13:20:43.953754 29202 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:43.954735 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1745) took 1722 ms. Trace:
I20240426 13:20:44.012595 27911 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:43.919631 29230 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:43.956302 29035 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.956421 29130 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:43.957875 29051 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.958292 29110 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.941640 29120 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:43.959450 29218 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:43.921512 29058 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:43.959676 28906 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:43.950357 28745 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:43.961120 28777 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:43.996681 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 726) took 1406 ms. Trace:
I20240426 13:20:44.049367 24956 rpcz_store.cc:276] 0426 13:20:42.590088 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.947375 (+1357287us) service_pool.cc:225] Handling call
0426 13:20:43.985555 (+ 38180us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.991807 (+  6252us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.996610 (+  4803us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.020915 29114 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:43.946116 29094 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:44.006711 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.004s	sys 0.014s
I20240426 13:20:44.007925 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:44.066411 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.004s	sys 0.014s
I20240426 13:20:44.067224 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.067503 29220 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.067968 25026 raft_consensus.cc:2354] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
W20240426 13:20:44.000356 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.021s	sys 0.032s
W20240426 13:20:44.068397 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.022s	sys 0.032s
W20240426 13:20:44.068445 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.046s	sys 0.036s
W20240426 13:20:44.068802 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.047s	sys 0.036s
W20240426 13:20:44.069047 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.31241e+06us
I20240426 13:20:43.946259 29103 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:43.952111 29105 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:44.019424 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.012s	sys 0.042s
W20240426 13:20:44.088599 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 679521us
I20240426 13:20:44.047884 29058 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:44.053118 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.006s	sys 0.018s
W20240426 13:20:44.098654 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.006s	sys 0.018s
I20240426 13:20:44.098069 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.099808 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:44.020285 25105 rpcz_store.cc:276] 0426 13:20:42.232247 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.354489 (+122242us) service_pool.cc:225] Handling call
0426 13:20:42.368940 (+ 14451us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:42.369146 (+   206us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:42.369170 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:42.382122 (+ 12952us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.803094 (+1420972us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:43.803147 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.803246 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:43.954530 (+151284us) raft_consensus.cc:1671] finished
0426 13:20:43.954562 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.954683 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1328263,"threads_started":1,"wal-append.queue_time_us":1468536}
W20240426 13:20:44.101039 28999 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.016s	sys 0.017s
I20240426 13:20:44.024807 24955 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:44.101454 28999 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.016s	sys 0.017s
I20240426 13:20:44.102048 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2363) took 1496 ms. Trace:
I20240426 13:20:44.102464 24955 rpcz_store.cc:276] 0426 13:20:42.605708 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.011549 (+1405841us) service_pool.cc:225] Handling call
0426 13:20:44.024641 (+ 13092us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.101758 (+ 77117us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.101987 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:44.106426 29009 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.008s	sys 0.021s
W20240426 13:20:44.107755 29009 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.009s	sys 0.022s
W20240426 13:20:44.025251 25025 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.009s	sys 0.011s
W20240426 13:20:44.011332 28626 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.007s	sys 0.010s
I20240426 13:20:44.011435 24947 rpcz_store.cc:276] 0426 13:20:40.371590 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.255327 (+883737us) service_pool.cc:225] Handling call
0426 13:20:41.255801 (+   474us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:41.940577 (+684776us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:41.940623 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:41.941379 (+   756us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.340451 (+1399072us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:43.456442 (+115991us) log.cc:842] Serialized 14 byte log entry
0426 13:20:43.456677 (+   235us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.456827 (+   150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:43.952506 (+495679us) raft_consensus.cc:1671] finished
0426 13:20:43.952529 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:43.952671 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":126789,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":13079,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7077,"thread_start_us":2065858,"threads_started":2,"wal-append.queue_time_us":1612036}
W20240426 13:20:44.163635 25025 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.009s	sys 0.013s
I20240426 13:20:44.165244 24947 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:44.165881 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2371) took 1492 ms. Trace:
I20240426 13:20:44.004675 29176 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:43.941082 28822 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.010937 28917 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.051101 24956 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:43.946465 29074 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.020220 29008 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:44.012881 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.000s	sys 0.016s
W20240426 13:20:44.053844 28998 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.000s	sys 0.043s
I20240426 13:20:44.122579 25026 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:44.049158 28777 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:44.126950 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:44.063450 29181 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.004s	sys 0.018s
I20240426 13:20:44.097514 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.127715 29063 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
W20240426 13:20:44.132352 29236 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20240426 13:20:44.133088 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:44.134548 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:44.135218 29234 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20240426 13:20:44.185113 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58276us
I20240426 13:20:44.185968 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.186198 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.186592 25105 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:44.187276 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.187390 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.135567 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2349) took 3749 ms. Trace:
I20240426 13:20:44.188177 24953 rpcz_store.cc:276] 0426 13:20:40.386479 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.258449 (+871970us) service_pool.cc:225] Handling call
0426 13:20:41.258870 (+   421us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:41.758124 (+499254us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:43.251487 (+1493363us) log.cc:842] Serialized 14 byte log entry
0426 13:20:43.626585 (+375098us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:43.824293 (+197708us) log.cc:842] Serialized 52 byte log entry
0426 13:20:43.824470 (+   177us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:43.825924 (+  1454us) log.cc:842] Serialized 14 byte log entry
0426 13:20:43.826120 (+   196us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.826186 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:44.135333 (+309147us) raft_consensus.cc:1671] finished
0426 13:20:44.135356 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.135510 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":270957,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":10680,"raft.queue_time_us":11977,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":90711,"thread_start_us":2251633,"threads_started":3,"wal-append.queue_time_us":775639}
I20240426 13:20:44.135871 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:44.137197 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:44.161527 29179 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.141482 29253 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:44.142169 28745 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:44.010744 29204 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.067210 29035 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.123447 29037 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20240426 13:20:44.089476 29218 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.122110 29105 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.089293 29110 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.157944 29247 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:44.159219 29148 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.090410 29050 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.088163 29197 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.029552 29130 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.028988 29120 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.023185 29005 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.166847 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2479) took 3133 ms. Trace:
I20240426 13:20:44.048118 29051 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:44.167687 25030 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.009s	sys 0.024s
I20240426 13:20:44.166224 24947 rpcz_store.cc:276] 0426 13:20:42.673409 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.164677 (+1491268us) service_pool.cc:225] Handling call
0426 13:20:44.165034 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.165656 (+   622us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.165830 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.090626 29220 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.173604 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2365) took 1504 ms. Trace:
I20240426 13:20:44.259902 24956 rpcz_store.cc:276] 0426 13:20:42.668950 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.050389 (+1381439us) service_pool.cc:225] Handling call
0426 13:20:44.050750 (+   361us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.171682 (+120932us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.173533 (+  1851us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:44.173267 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.000s	sys 0.018s
I20240426 13:20:44.172607 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:44.177248 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 264047us
I20240426 13:20:44.150776 28808 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20240426 13:20:44.264041 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.182865 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.166678 29045 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:44.171509 29006 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.920s	user 0.010s	sys 0.001s
I20240426 13:20:44.189505 25104 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.196368 29016 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:44.190697 25100 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.195793 29176 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:44.140149 28900 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:44.096693 29103 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.172853 29241 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:44.191540 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94094.1us
I20240426 13:20:44.300046 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:44.176000 28998 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.000s	sys 0.043s
W20240426 13:20:44.178083 29181 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.004s	sys 0.018s
I20240426 13:20:44.192355 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.305820 25103 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:44.171507 29250 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.294662 29016 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:44.310477 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:44.194162 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:44.194398 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:44.181892 28777 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:44.195029 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 787) took 1894 ms. Trace:
I20240426 13:20:44.212505 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2373) took 1513 ms. Trace:
I20240426 13:20:44.257323 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.251089 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2494) took 3147 ms. Trace:
W20240426 13:20:44.258433 25030 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.011s	sys 0.024s
I20240426 13:20:44.239264 25025 rpcz_store.cc:276] 0426 13:20:41.033507 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.359220 (+2325713us) service_pool.cc:225] Handling call
0426 13:20:43.419822 (+ 60602us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.165954 (+746132us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:44.166111 (+   157us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.166229 (+   118us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:44.166311 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.166500 (+   189us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.166720 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":405883,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":13169,"thread_start_us":745042,"threads_started":1}
I20240426 13:20:44.260952 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.195360 28917 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:44.179455 29008 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.261428 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2115) took 1877 ms. Trace:
I20240426 13:20:44.266835 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:44.268646 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.190459 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:44.237135 29111 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.985s	user 0.009s	sys 0.003s
I20240426 13:20:44.196322 29010 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.295153 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.221927 29081 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.223896 28745 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:44.198359 29063 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.219224 29148 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:44.208431 29255 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:44.260327 29247 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.307116 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.193230 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.206777 29035 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.207598 29110 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.207365 29218 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.295135 29186 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.330554 25024 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:44.221153 29050 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.291478 29252 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:44.360404 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 166305us
I20240426 13:20:44.332700 24953 rpcz_store.cc:276] 0426 13:20:42.699095 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.189989 (+1490894us) service_pool.cc:225] Handling call
0426 13:20:44.211791 (+ 21802us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.212017 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:44.212088 (+    71us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.212145 (+    57us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:44.212198 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.212264 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.212434 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.311596 29217 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:44.314769 25100 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:44.366449 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2493) took 3262 ms. Trace:
I20240426 13:20:44.366518 25026 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:44.295228 29045 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:44.367153 29254 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20240426 13:20:44.384740 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:44.387777 29256 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
W20240426 13:20:44.372545 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105791us
I20240426 13:20:44.330708 29009 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.332558 25619 rpcz_store.cc:276] 0426 13:20:42.300400 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.636731 (+336331us) service_pool.cc:225] Handling call
0426 13:20:42.637196 (+   465us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.677767 (+ 40571us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:43.872836 (+1195069us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.194561 (+321725us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.194619 (+    58us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:44.194683 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.194815 (+   132us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.194960 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8707,"prepare.queue_time_us":1342431,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":39,"raft.queue_time_us":15628,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3664,"thread_start_us":1438366,"threads_started":2,"wal-append.queue_time_us":420777}
W20240426 13:20:44.393401 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 124861us
I20240426 13:20:44.363227 25102 rpcz_store.cc:276] 0426 13:20:42.383731 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.483317 (+ 99586us) service_pool.cc:225] Handling call
0426 13:20:42.483719 (+   402us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:42.483970 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:42.484003 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:42.541205 (+ 57202us) log.cc:842] Serialized 26 byte log entry
0426 13:20:43.865154 (+1323949us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:43.865197 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:43.865265 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:44.261218 (+395953us) raft_consensus.cc:1671] finished
0426 13:20:44.261246 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.261379 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1316183,"threads_started":1,"wal-append.queue_time_us":1400756}
I20240426 13:20:44.333043 29230 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.370841 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2466) took 3468 ms. Trace:
I20240426 13:20:44.302443 29241 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.451409 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.333753 25023 rpcz_store.cc:276] 0426 13:20:41.103819 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.822002 (+2718183us) service_pool.cc:225] Handling call
0426 13:20:43.847183 (+ 25181us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.847406 (+   223us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:44.107781 (+260375us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.250578 (+142797us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.250650 (+    72us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:44.250717 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.250809 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.250988 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33032,"prepare.queue_time_us":382187,"prepare.run_wall_time_us":6845,"raft.queue_time_us":74308,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":5362,"spinlock_wait_cycles":11847808,"thread_start_us":367474,"threads_started":2,"wal-append.queue_time_us":252261}
I20240426 13:20:44.448617 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:44.452544 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.503533 25100 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:44.442224 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 249094us
I20240426 13:20:44.451359 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:44.451192 29273 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20240426 13:20:44.446321 29276 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:20:44.454948 29262 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20240426 13:20:44.455076 29050 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.455269 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.462088 25029 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
W20240426 13:20:44.502472 28429 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.937s	user 0.007s	sys 0.003s
I20240426 13:20:44.448452 25027 rpcz_store.cc:276] 0426 13:20:41.103482 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.821800 (+2718318us) service_pool.cc:225] Handling call
0426 13:20:43.861827 (+ 40027us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.862078 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:44.154132 (+292054us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.364064 (+209932us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.364502 (+   438us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:44.365066 (+   564us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.365398 (+   332us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.366008 (+   610us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49077,"prepare.queue_time_us":403208,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2350,"raft.queue_time_us":66213,"raft.run_wall_time_us":3021,"spinlock_wait_cycles":20726144,"thread_start_us":451549,"threads_started":3,"wal-append.queue_time_us":212427}
I20240426 13:20:44.444767 28917 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.400722 29081 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.465199 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.450160 29148 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.469475 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.449465 29244 raft_consensus.cc:2741] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:20:44.447423 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:44.472535 29105 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.479187 29103 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.452723 29008 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.480794 29252 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:44.372197 29246 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.118s
I20240426 13:20:44.485414 29130 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.485798 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1266) took 1116 ms. Trace:
I20240426 13:20:44.486223 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 788) took 1944 ms. Trace:
I20240426 13:20:44.546460 25621 rpcz_store.cc:276] 0426 13:20:42.541826 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.259951 (+718125us) service_pool.cc:225] Handling call
0426 13:20:43.353023 (+ 93072us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.851434 (+498411us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.485998 (+634564us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.486170 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1078938496}
I20240426 13:20:44.486795 29220 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.487731 29120 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.471975 28870 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20240426 13:20:44.488603 29134 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:44.491492 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 754) took 1824 ms. Trace:
I20240426 13:20:44.491817 28777 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.494457 29204 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.495596 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 704) took 1410 ms. Trace:
W20240426 13:20:44.499758 29193 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.556s	user 0.003s	sys 0.005s
I20240426 13:20:44.449615 28822 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.501138 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1753) took 3932 ms. Trace:
I20240426 13:20:44.560187 25031 rpcz_store.cc:276] 0426 13:20:40.568775 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:41.388569 (+819794us) service_pool.cc:225] Handling call
0426 13:20:44.500956 (+3112387us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.501086 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5232974208}
I20240426 13:20:44.561554 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.562204 25031 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.501629 25022 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:44.472752 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:44.333364 29250 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.464749 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:44.566087 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101416us
I20240426 13:20:44.566860 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:44.471508 25030 rpcz_store.cc:276] 0426 13:20:40.902886 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.345768 (+1442882us) service_pool.cc:225] Handling call
0426 13:20:42.360873 (+ 15105us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:42.361134 (+   261us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:42.378059 (+ 16925us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.364125 (+1986066us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.364478 (+   353us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:44.364538 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.369564 (+  5026us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.370784 (+  1220us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18304,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":406111,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":33413,"thread_start_us":1983640,"threads_started":2,"wal-append.queue_time_us":1370676}
I20240426 13:20:44.367262 29111 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.525513 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.368762 29037 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:44.549458 29253 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:44.528403 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:44.529085 25105 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
I20240426 13:20:44.530138 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.531451 25026 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:44.466120 29247 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
W20240426 13:20:44.574045 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48603.9us
I20240426 13:20:44.575145 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:44.532562 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:44.533342 25102 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:44.572814 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2500) took 3162 ms. Trace:
I20240426 13:20:44.576452 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.574070 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 550) took 3104 ms. Trace:
I20240426 13:20:44.449883 29186 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.533072 24953 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:44.543136 25027 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:44.471061 29009 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.554499 29264 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20240426 13:20:44.527709 29243 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:44.544685 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97341.8us
I20240426 13:20:44.546403 25101 rpcz_store.cc:276] 0426 13:20:43.369524 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.399904 (+ 30380us) service_pool.cc:225] Handling call
0426 13:20:44.485346 (+1085442us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.485734 (+   388us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037623168}
I20240426 13:20:44.569052 29246 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.554468 25615 rpcz_store.cc:276] 0426 13:20:42.667211 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.356899 (+689688us) service_pool.cc:225] Handling call
0426 13:20:43.438334 (+ 81435us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.576062 (+137728us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.491281 (+915219us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.491442 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":298641280}
I20240426 13:20:44.533082 29179 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
W20240426 13:20:44.550451 29275 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20240426 13:20:44.534111 29126 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20240426 13:20:44.556907 25096 rpcz_store.cc:276] 0426 13:20:43.085468 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.410650 (+325182us) service_pool.cc:225] Handling call
0426 13:20:43.465872 (+ 55222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.946710 (+480838us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:43.946747 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:43.946805 (+    58us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:43.946858 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.495383 (+548525us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.495428 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.495527 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105346944}
I20240426 13:20:44.602567 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 723) took 2052 ms. Trace:
I20240426 13:20:44.590878 24949 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:44.581035 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:44.600914 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1744) took 2369 ms (client timeout 2172 ms). Trace:
I20240426 13:20:44.585832 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:44.604732 25099 rpcz_store.cc:269] 0426 13:20:42.231575 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.346697 (+115122us) service_pool.cc:225] Handling call
0426 13:20:44.600732 (+2254035us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.600849 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3412791040}
I20240426 13:20:44.563261 24956 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:44.568347 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20240426 13:20:44.606846 24954 pending_rounds.cc:76] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:20:44.607270 24954 pending_rounds.cc:97] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20240426 13:20:44.607656 24954 raft_consensus.cc:2866] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:44.536523 29181 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20240426 13:20:44.567834 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:44.571276 29037 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:44.569254 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2486) took 3533 ms. Trace:
I20240426 13:20:44.528364 25029 pending_rounds.cc:76] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 5
I20240426 13:20:44.571627 25104 raft_consensus.cc:2354] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11.
I20240426 13:20:44.534839 28917 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.608598 25022 rpcz_store.cc:276] 0426 13:20:41.035576 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.449074 (+2413498us) service_pool.cc:225] Handling call
0426 13:20:44.501461 (+1052387us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.563591 (+ 62130us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.564561 (+   970us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400242a50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.569205 (+  4644us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2419592576}
I20240426 13:20:44.608704 25029 pending_rounds.cc:97] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20240426 13:20:44.609886 25029 raft_consensus.cc:2866] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:44.609968 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.610262 28917 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.592377 29243 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.552389 29252 consensus_queue.cc:1225] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:44.554373 29134 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.611080 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.577219 25026 rpcz_store.cc:276] 0426 13:20:41.469820 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.530914 (+3061094us) service_pool.cc:225] Handling call
0426 13:20:44.531169 (+   255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.573913 (+ 42744us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.574035 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.536336 29245 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20240426 13:20:44.595206 25101 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:44.580825 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2501) took 3133 ms. Trace:
I20240426 13:20:44.617954 25027 rpcz_store.cc:276] 0426 13:20:41.447809 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.530057 (+3082248us) service_pool.cc:225] Handling call
0426 13:20:44.542859 (+ 12802us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:44.580594 (+ 37735us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.580781 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.533046 29084 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:44.596628 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.590283 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2117) took 1336 ms. Trace:
I20240426 13:20:44.576469 28870 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:44.610939 29017 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:44.588723 29110 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.580778 29279 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:44.580572 29005 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.577638 29247 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.585579 29009 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.596854 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:44.573474 29114 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.511s	user 0.008s	sys 0.000s
I20240426 13:20:44.541150 29010 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20240426 13:20:44.571130 29218 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.640264 29134 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.625123 29084 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.640796 29247 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.612237 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:44.612408 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 418067us
I20240426 13:20:44.612871 25102 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
W20240426 13:20:44.612408 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 439855us
I20240426 13:20:44.649312 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.579680 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.602946 24948 rpcz_store.cc:276] 0426 13:20:42.550515 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.795708 (+1245193us) service_pool.cc:225] Handling call
0426 13:20:43.796073 (+   365us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:43.851648 (+ 55575us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:43.851676 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:43.854739 (+  3063us) log.cc:842] Serialized 52 byte log entry
0426 13:20:44.166616 (+311877us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:44.174459 (+  7843us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.589422 (+414963us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b1800518920. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:20:44.589590 (+   168us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.589668 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:44.594833 (+  5165us) raft_consensus.cc:1671] finished
0426 13:20:44.602393 (+  7560us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.602519 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57427,"prepare.queue_time_us":9392,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":491945,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":33781,"spinlock_wait_cycles":16853760,"thread_start_us":243939,"threads_started":1,"wal-append.queue_time_us":664315}
I20240426 13:20:44.642057 29008 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:44.650976 29008 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.651371 24948 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [7.4-8.5]
I20240426 13:20:44.651974 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2375) took 1927 ms. Trace:
I20240426 13:20:44.652314 24948 rpcz_store.cc:276] 0426 13:20:42.725037 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.650841 (+1925804us) service_pool.cc:225] Handling call
0426 13:20:44.651158 (+   317us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:44.651754 (+   596us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.651931 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.653477 24948 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20240426 13:20:44.653200 29208 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.653875 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:44.612130 29035 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.654565 29197 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:44.655925 29197 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:44.578261 29253 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:44.615275 29243 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.612718 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:44.656708 29197 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:44.657058 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1941) took 2850 ms. Trace:
I20240426 13:20:44.657178 29197 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:44.615470 29204 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:44.576879 28822 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:44.656736 29253 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.658043 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.581112 29120 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:44.659173 25025 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:44.659869 24956 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:44.622696 25030 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:44.665194 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2368) took 1994 ms. Trace:
I20240426 13:20:44.665604 24955 rpcz_store.cc:276] 0426 13:20:42.670959 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.103303 (+1432344us) service_pool.cc:225] Handling call
0426 13:20:44.103647 (+   344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.235476 (+131829us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:44.235785 (+   309us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.612019 (+376234us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.612080 (+    61us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:44.612135 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.664956 (+ 52821us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.665004 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.665140 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":304608,"prepare.queue_time_us":64410,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":307857,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":302479104,"thread_start_us":71264,"threads_started":1,"wal-append.queue_time_us":210921}
W20240426 13:20:44.667807 29272 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.167s	user 0.000s	sys 0.002s
I20240426 13:20:44.668779 28626 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:44.669304 28626 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.670320 28626 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:44.671689 29224 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.134s	user 0.003s	sys 0.003s
I20240426 13:20:44.671885 28626 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:44.675336 29258 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20240426 13:20:44.668783 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2380) took 1324 ms. Trace:
I20240426 13:20:44.676826 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 721) took 2136 ms. Trace:
W20240426 13:20:44.677788 29058 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s	user 0.004s	sys 0.004s
I20240426 13:20:44.678346 29058 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.534721 29063 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.592522 29094 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:44.679777 29094 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:44.680325 29094 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.600749 29016 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.684667 29006 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:44.685165 29006 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:44.686131 29241 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.608561 29246 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.688200 25096 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:44.592327 29148 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.589776 29111 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.591734 28777 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.630368 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.612305 29037 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.690001 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 757) took 2015 ms. Trace:
I20240426 13:20:44.690778 29030 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:44.616580 29264 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:20:44.600345 29250 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.648640 29247 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:44.622843 25097 rpcz_store.cc:276] 0426 13:20:43.253849 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.355373 (+101524us) service_pool.cc:225] Handling call
0426 13:20:43.355678 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.882235 (+526557us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.590074 (+707839us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.590229 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1137280256}
I20240426 13:20:44.614787 29193 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:44.686175 29006 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.697820 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2464) took 3863 ms. Trace:
I20240426 13:20:44.698050 29006 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:44.698200 25024 rpcz_store.cc:276] 0426 13:20:40.848299 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.328695 (+1480396us) service_pool.cc:225] Handling call
0426 13:20:42.361589 (+ 32894us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.445171 (+2083582us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.697576 (+252405us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.697762 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4482726272}
I20240426 13:20:44.699407 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.690397 25624 rpcz_store.cc:276] 0426 13:20:42.674195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.406715 (+732520us) service_pool.cc:225] Handling call
0426 13:20:43.448669 (+ 41954us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.830414 (+381745us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:43.830459 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:43.830517 (+    58us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:43.830563 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.689796 (+859233us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.689842 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.689952 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":877334272}
I20240426 13:20:44.656333 29045 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.610895 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2499) took 3211 ms. Trace:
I20240426 13:20:44.703133 25029 rpcz_store.cc:276] 0426 13:20:41.399283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.461487 (+3062204us) service_pool.cc:225] Handling call
0426 13:20:44.461768 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.610694 (+148926us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.610848 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.658435 29204 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:44.659916 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.704579 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.661672 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 782) took 3006 ms. Trace:
I20240426 13:20:44.676543 24955 rpcz_store.cc:276] 0426 13:20:43.344609 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.667771 (+1323162us) service_pool.cc:225] Handling call
0426 13:20:44.668130 (+   359us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.668335 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:44.668372 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.668427 (+    55us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:44.668471 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.668547 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.668658 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.677439 24950 rpcz_store.cc:276] 0426 13:20:42.540638 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.566637 (+1025999us) service_pool.cc:225] Handling call
0426 13:20:43.587096 (+ 20459us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.597350 (+ 10254us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:43.597393 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:43.760726 (+163333us) log.cc:842] Serialized 26 byte log entry
0426 13:20:44.185183 (+424457us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:44.312496 (+127313us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.480507 (+168011us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:20:44.596085 (+115578us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.596158 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:44.674869 (+ 78711us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:44.676648 (+  1779us) raft_consensus.cc:1671] finished
0426 13:20:44.676666 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.676776 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":412067,"prepare.queue_time_us":29866,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":8551,"raft.queue_time_us":300870,"raft.run_cpu_time_us":3007,"raft.run_wall_time_us":815509,"spinlock_wait_cycles":77594752,"thread_start_us":464153,"threads_started":2,"wal-append.queue_time_us":713468}
I20240426 13:20:44.648847 29279 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:44.707075 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.707610 25024 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:44.654203 29208 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.641302 29134 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.641963 29114 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:44.666126 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.725631 25030 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
W20240426 13:20:44.725768 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146193us
I20240426 13:20:44.620738 25589 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
I20240426 13:20:44.657481 24952 rpcz_store.cc:276] 0426 13:20:41.806284 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.547963 (+1741679us) service_pool.cc:225] Handling call
0426 13:20:43.548345 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:43.878763 (+330418us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.656857 (+778094us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b540004c254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.657009 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":690904960}
I20240426 13:20:44.726701 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.704712 25025 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
W20240426 13:20:44.727447 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77801.9us
W20240426 13:20:44.727761 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78579.5us
I20240426 13:20:44.728161 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.728514 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.705339 25622 rpcz_store.cc:276] 0426 13:20:41.654999 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:42.537820 (+882821us) service_pool.cc:225] Handling call
0426 13:20:42.538179 (+   359us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.927197 (+1389018us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:43.927228 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:43.927267 (+    39us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:43.927296 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.661451 (+734155us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b5400473954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.661497 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.661618 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21431,"raft.queue_time_us":130865,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":2004453248,"thread_start_us":495543,"threads_started":1}
I20240426 13:20:44.660938 25100 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:44.729542 25100 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.701746 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2379) took 1357 ms. Trace:
I20240426 13:20:44.654048 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2377) took 1832 ms. Trace:
I20240426 13:20:44.630009 29017 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:44.727082 25589 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
W20240426 13:20:44.737488 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 131800us
I20240426 13:20:44.619202 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.738003 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.731092 24956 rpcz_store.cc:276] 0426 13:20:43.344377 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.659436 (+1315059us) service_pool.cc:225] Handling call
0426 13:20:44.659711 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.701529 (+ 41818us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.701696 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.735651 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:44.739805 25621 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:44.699913 29181 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:44.745451 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.746832 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.748263 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.731504 24948 rpcz_store.cc:276] 0426 13:20:42.821706 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.652990 (+1831284us) service_pool.cc:225] Handling call
0426 13:20:44.653283 (+   293us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:44.653829 (+   546us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.654003 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:44.730435 25100 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.750388 25100 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:44.750788 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.751416 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.752388 25022 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:44.753386 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.753907 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.762403 24948 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 13:20:44.789304 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.789304 24950 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:44.791563 25025 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:44.789667 25615 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:44.755138 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.755890 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:44.768448 25621 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:44.762602 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.793498 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2492) took 3690 ms. Trace:
I20240426 13:20:44.754746 25024 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:44.774480 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.797340 25028 rpcz_store.cc:276] 0426 13:20:41.103179 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.552539 (+2449360us) service_pool.cc:225] Handling call
0426 13:20:43.567313 (+ 14774us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.849933 (+282620us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:44.194444 (+344511us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.502731 (+308287us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.502782 (+    51us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:44.502833 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.667259 (+164426us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:44.793263 (+126004us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.793322 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.793447 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20374,"prepare.queue_time_us":409918,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2328,"raft.queue_time_us":54979,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":29921,"spinlock_wait_cycles":653668352,"thread_start_us":592349,"threads_started":3,"wal-append.queue_time_us":306257}
I20240426 13:20:44.791836 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.796114 25619 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:44.801298 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.762652 25622 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:44.657974 29197 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.743165 29218 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:44.803622 29197 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.803483 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:44.793561 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.807878 24952 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:44.807826 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:44.744468 29245 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.743669 29250 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.651710 29008 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.742837 29017 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:44.791029 29058 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:44.815197 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60139.5us
I20240426 13:20:44.823518 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.822063 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:44.822376 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.832293 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.750061 29074 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.743979 29279 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.751175 29111 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:44.838359 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.750165 29005 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.753262 29120 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:44.790549 29130 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:44.742915 29246 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.752835 29114 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.790372 28822 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.793799 29243 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:20:44.814131 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 246390us
I20240426 13:20:44.743012 29181 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.825132 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.576625 25023 rpcz_store.cc:276] 0426 13:20:41.410700 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.504673 (+3093973us) service_pool.cc:225] Handling call
0426 13:20:44.532429 (+ 27756us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.532609 (+   180us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:44.534278 (+  1669us) log.cc:842] Serialized 14 byte log entry
0426 13:20:44.572443 (+ 38165us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.572480 (+    37us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:44.572513 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.572557 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:44.572656 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3346,"prepare.queue_time_us":1815,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":4579,"thread_start_us":34488,"threads_started":1}
I20240426 13:20:44.814230 28940 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:44.818233 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.850284 24956 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:44.823952 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:44.852203 29008 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.852959 25028 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:44.744199 29009 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.082s
I20240426 13:20:44.854084 29181 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.833387 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.835084 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:44.856159 25030 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:44.857803 29261 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:44.858287 29261 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:44.858915 29114 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.859081 29261 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.860419 25029 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:44.750418 29105 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:44.837298 25025 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:20:44.862114 29279 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.863343 29181 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:44.863574 29186 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:44.824743 29193 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:44.865350 29205 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:44.865857 29051 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:44.867026 29252 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:44.867376 29252 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.842171 25100 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:44.753043 29122 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:44.869208 29122 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:44.869572 29122 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.855859 28899 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20240426 13:20:44.845445 29250 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.872468 28899 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:44.872804 28899 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:44.873468 28899 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:44.873979 29269 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.373s	user 0.000s	sys 0.002s
I20240426 13:20:44.874567 28899 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:44.875293 28906 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:44.875628 28906 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:44.831331 29204 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.879586 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.840278 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.825723 29278 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:44.883585 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 725) took 2332 ms. Trace:
I20240426 13:20:44.884004 24954 rpcz_store.cc:276] 0426 13:20:42.551222 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.928535 (+1377313us) service_pool.cc:225] Handling call
0426 13:20:43.928955 (+   420us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.608214 (+679259us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.883380 (+275166us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:44.883525 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1462302976}
I20240426 13:20:44.889288 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:44.891149 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 705) took 1483 ms. Trace:
I20240426 13:20:44.891563 25098 rpcz_store.cc:276] 0426 13:20:43.407615 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.407795 (+   180us) service_pool.cc:225] Handling call
0426 13:20:44.891008 (+1483213us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.891099 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":884809088}
I20240426 13:20:44.896606 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.897800 25030 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:44.895677 25027 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:44.902252 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:44.883141 29081 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20240426 13:20:44.912643 25023 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:44.909416 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1929) took 3139 ms. Trace:
I20240426 13:20:44.913444 24951 rpcz_store.cc:276] 0426 13:20:41.769774 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.468434 (+1698660us) service_pool.cc:225] Handling call
0426 13:20:43.827305 (+358871us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:43.846535 (+ 19230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:44.909165 (+1062630us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54003683d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.909328 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":680305920}
W20240426 13:20:44.910512 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2376) took 2183 ms (client timeout 1857 ms). Trace:
W20240426 13:20:44.915287 24949 rpcz_store.cc:269] 0426 13:20:42.726741 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:43.265678 (+538937us) service_pool.cc:225] Handling call
0426 13:20:44.910270 (+1644592us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001018d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:44.910437 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3045160192}
I20240426 13:20:44.828887 29110 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20240426 13:20:44.903862 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.879042 29120 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.878682 29267 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:44.925333 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:44.926743 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.939031 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.929481 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:44.930322 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.930873 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:44.931551 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:44.934885 25623 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:44.897465 29009 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20240426 13:20:44.888177 29205 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:44.936863 24949 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [9.8-11.9]
I20240426 13:20:44.940436 24956 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
W20240426 13:20:44.937165 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97006.1us
I20240426 13:20:44.944589 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:44.944839 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.900482 29278 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.945600 25098 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.949340 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.950379 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.940714 24952 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:44.940939 25621 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:44.951077 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:44.904789 29193 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:44.883522 29005 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20240426 13:20:44.947383 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:44.951906 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:44.952440 24956 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:44.908115 29090 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:44.952481 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:44.953366 25098 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.907600 29105 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:44.957639 25621 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:20:44.950706 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:44.974390 25615 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:44.951212 25101 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.874017 29253 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.975272 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:44.968451 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:44.976681 25620 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:20:44.978478 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.936136 29130 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:44.968449 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.968943 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:44.948695 29220 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240426 13:20:44.971115 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 178269us
I20240426 13:20:44.973089 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:44.939924 29243 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:44.982399 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:44.906530 29111 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20240426 13:20:44.968971 25031 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:44.983985 25097 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.880334 29252 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.938875 29282 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20240426 13:20:44.972577 25096 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:44.991858 25102 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:44.987406 29066 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20240426 13:20:45.051565 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.965262 29267 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:44.965096 29017 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:44.940661 28906 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.972820 29090 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:45.005614 29030 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:45.063616 29267 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:45.058408 29278 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:44.965306 29037 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.021781 29281 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:44.986644 29220 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:45.074620 29017 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.078300 29218 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
I20240426 13:20:45.080662 29252 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.091745 29220 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.114630 29253 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.107331 29267 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.121556 29267 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:45.124179 29250 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:45.091435 29281 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.127039 29281 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.157696 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:45.160893 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 243263us
I20240426 13:20:45.162881 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.165902 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.170867 24921 leader_election.cc:398] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:44.941021 24948 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:44.970923 24954 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:45.301213 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 130734us
I20240426 13:20:45.303555 24921 leader_election.cc:398] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:45.303900 24921 leader_election.cc:401] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240426 13:20:45.306314 29261 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
W20240426 13:20:45.172508 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 189134us
I20240426 13:20:45.173511 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2374) took 2471 ms. Trace:
I20240426 13:20:45.173898 25023 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.308961 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.309870 29261 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.173290 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:45.022897 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:45.180888 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:45.313370 29176 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:45.313928 25024 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.182621 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.024683 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:44.987399 25104 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:45.186956 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.341251 25098 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:45.341779 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:45.342624 25104 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 10
W20240426 13:20:45.188009 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 373656us
I20240426 13:20:45.344933 24952 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:45.345975 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 321254us
I20240426 13:20:45.189811 25096 pending_rounds.cc:76] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 8
I20240426 13:20:45.346146 29244 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:45.346629 25096 pending_rounds.cc:97] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20240426 13:20:45.347049 25096 raft_consensus.cc:2866] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:45.190631 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 139032us
I20240426 13:20:45.349027 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.350265 29081 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:45.198515 28899 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:45.350764 29281 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:45.019817 28745 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20240426 13:20:45.199443 29005 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:45.352393 29283 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:45.352892 29283 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.190712 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.353212 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:45.354007 29283 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.354763 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.210915 29111 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:45.111296 29238 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:45.404814 29238 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:45.406379 29238 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.102671 29243 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:45.408768 29008 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.556s	user 0.006s	sys 0.013s
I20240426 13:20:45.410094 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.215096 29037 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.412856 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.414216 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.415280 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.218248 29084 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.219820 25022 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:45.038444 29120 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.227325 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.244457 24950 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:45.219126 29090 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.248648 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.227840 29122 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.312413 29066 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:45.431762 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.331796 25624 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:20:45.333540 29094 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.652s	user 0.013s	sys 0.003s
I20240426 13:20:45.433357 29094 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.433811 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:45.434638 25097 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.435601 29094 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:45.336503 29105 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:45.440865 29105 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:45.441229 29105 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:45.442191 29105 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.443892 25105 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:45.443364 29005 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:45.444403 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4139181259999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20240426 13:20:45.337177 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.339618 25620 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:45.350337 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.209829 29017 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:45.446677 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1700181260000591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21720000000000003s for a response
I20240426 13:20:45.308432 24953 rpcz_store.cc:276] 0426 13:20:42.701408 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.580769 (+1879361us) service_pool.cc:225] Handling call
0426 13:20:44.601196 (+ 20427us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:44.616195 (+ 14999us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:44.616257 (+    62us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:44.628119 (+ 11862us) log.cc:842] Serialized 52 byte log entry
0426 13:20:44.940322 (+312203us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:45.172091 (+231769us) log.cc:842] Serialized 14 byte log entry
0426 13:20:45.172678 (+   587us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.172864 (+   186us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:45.172993 (+   129us) raft_consensus.cc:1671] finished
0426 13:20:45.173023 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:45.173248 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14437,"prepare.queue_time_us":300085,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":85,"raft.queue_time_us":59916,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":417,"thread_start_us":248043,"threads_started":1,"wal-append.queue_time_us":380533}
I20240426 13:20:45.341522 29253 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20240426 13:20:45.356297 29283 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.451057 25104 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:45.369359 29261 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:45.369663 29293 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:45.371431 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:45.372381 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.373809 25615 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:20:45.375484 25028 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:45.375447 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:45.375998 29208 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.631s	user 0.012s	sys 0.007s
I20240426 13:20:45.376660 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:45.380363 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 357413us
W20240426 13:20:45.381590 29074 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.540s	user 0.010s	sys 0.003s
I20240426 13:20:45.382587 28906 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:45.385713 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.387116 28870 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:45.472781 25622 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:45.400092 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.370919 25024 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:45.374483 29205 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:45.396700 29220 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.411629 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.417225 25029 raft_consensus.cc:2396] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20240426 13:20:45.420763 25023 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:45.087000 29030 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:45.365424 29307 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
I20240426 13:20:45.433355 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:45.426808 29238 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.451329 25097 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:45.444734 28758 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:45.443490 29105 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:45.462881 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47420.8us
I20240426 13:20:45.396531 29252 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:45.468667 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.533766 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.472822 28870 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.535295 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.501070 25616 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:45.535761 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.000s	sys 0.014s
W20240426 13:20:45.536538 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 701478us
I20240426 13:20:45.537017 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:45.537506 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.537786 25030 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:45.539824 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 714) took 1083 ms. Trace:
I20240426 13:20:45.540261 29051 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:45.500546 25618 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.542393 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.542629 24949 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:45.543246 25025 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:45.544250 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.456565 29293 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:45.544950 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.545264 29293 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:45.545773 25029 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:45.545662 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:45.546499 25102 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:45.547014 25029 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20240426 13:20:45.547659 25102 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:45.533917 29122 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.548381 29208 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240426 13:20:45.548648 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 198481us
I20240426 13:20:45.549326 25103 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:45.550145 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.510221 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.569885 29090 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.570386 29043 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20240426 13:20:45.570772 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106474us
I20240426 13:20:45.571209 25615 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20240426 13:20:45.510783 28899 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.551555 29148 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:45.537667 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.551591 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.573220 25621 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:45.512104 29074 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:45.553570 29017 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:45.574283 29074 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:45.575004 29122 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:45.554162 29111 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:45.531121 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.576169 29179 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:45.576345 25026 raft_consensus.cc:2362] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10. Re-sending same reply.
I20240426 13:20:45.576805 29179 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.554530 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.540364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:45.535347 24954 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:45.578985 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1947) took 1385 ms. Trace:
I20240426 13:20:45.535322 28870 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:45.549346 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.909s (SENT) [suppressed 25 similar messages]
I20240426 13:20:45.544984 25623 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:45.579380 24955 rpcz_store.cc:276] 0426 13:20:44.193910 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.708321 (+514411us) service_pool.cc:225] Handling call
0426 13:20:44.750058 (+ 41737us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:44.852282 (+102224us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:44.852310 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:44.852346 (+    36us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:44.852378 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.464562 (+612184us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b54000794d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:45.467712 (+  3150us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:45.578820 (+111108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5440,"raft.queue_time_us":9989,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"spinlock_wait_cycles":221850496}
I20240426 13:20:45.553566 29252 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20240426 13:20:45.540825 29051 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:45.581816 28870 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:45.581887 29051 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.557410 25023 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20240426 13:20:45.531826 28758 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.560871 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:45.530874 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56057.1us
I20240426 13:20:45.561100 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.560878 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.539758 29220 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.563247 29297 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.572264 29205 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:45.572779 29148 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.621537 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:45.621870 29030 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.621881 29148 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.573557 25024 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:45.623520 29148 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.552305 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:45.624279 25101 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:45.624439 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:45.624742 25101 raft_consensus.cc:603] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:45.578166 29084 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:45.625751 25588 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.626686 29043 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:45.626149 28758 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.572784 25615 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20240426 13:20:45.627398 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55238.6us
I20240426 13:20:45.627694 28758 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.615530 25023 pending_rounds.cc:76] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 9
I20240426 13:20:45.637571 25023 pending_rounds.cc:97] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20240426 13:20:45.570019 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.579941 28429 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:45.639441 25023 raft_consensus.cc:2866] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:45.542866 29006 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.844s	user 0.007s	sys 0.012s
I20240426 13:20:45.627070 29043 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:45.656204 29051 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:45.665022 25622 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:45.665073 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 95167.3us
I20240426 13:20:45.660051 29043 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:45.666442 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117100us
W20240426 13:20:45.671187 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 116801us
I20240426 13:20:45.531234 29005 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20240426 13:20:45.670104 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.676512 25619 raft_consensus.cc:2396] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:45.682133 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:45.677166 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.685901 29084 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:45.686364 29084 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.572795 28899 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.690532 25618 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:45.692669 24921 leader_election.cc:398] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8.
I20240426 13:20:45.650873 29297 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:45.700237 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53006.6us
I20240426 13:20:45.698746 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.703019 28777 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:45.703663 28777 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.704820 28777 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.705510 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.704839 29281 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20240426 13:20:45.709672 29297 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.723071 29218 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:45.723578 29218 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:45.723917 29218 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.692633 28429 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:45.729590 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49315.3us
I20240426 13:20:45.727908 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:45.730787 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58122.4us
I20240426 13:20:45.742075 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:45.751706 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74702.3us
I20240426 13:20:45.755029 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.744139 29006 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:45.729398 29081 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:45.763732 29220 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:45.692201 29084 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:45.769162 29130 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.768s	user 0.009s	sys 0.006s
I20240426 13:20:45.740482 29252 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:45.788069 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.765403 29006 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:45.804567 29006 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:45.804860 29009 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:45.807621 25622 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:45.795358 29084 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:45.820565 28429 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:45.821983 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 127191us
I20240426 13:20:45.822863 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.823607 29238 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:45.798398 29005 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:45.800554 25619 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20240426 13:20:45.801654 25618 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:45.816550 25589 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
I20240426 13:20:45.817565 29130 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:45.798071 29105 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.824638 29005 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.780880 29252 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.791149 25104 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:45.783031 29253 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:45.784715 29016 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.764441 29293 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:45.841563 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83443.2us
I20240426 13:20:45.842720 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 715) took 1335 ms. Trace:
W20240426 13:20:45.843060 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49419.8us
I20240426 13:20:45.847959 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1767) took 4217 ms. Trace:
I20240426 13:20:45.844228 25105 rpcz_store.cc:276] 0426 13:20:44.502203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.617348 (+115145us) service_pool.cc:225] Handling call
0426 13:20:44.627592 (+ 10244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:45.444441 (+816849us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.788801 (+344360us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400168854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:45.837213 (+ 48412us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1876487040}
I20240426 13:20:45.848367 25022 rpcz_store.cc:276] 0426 13:20:41.623492 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.824070 (+3200578us) service_pool.cc:225] Handling call
0426 13:20:44.824335 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:45.436444 (+612109us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.832880 (+396436us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b54004164d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:45.841188 (+  8308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":907993984}
I20240426 13:20:45.852524 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:45.859308 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 128799us
I20240426 13:20:45.842321 29220 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:45.867059 29176 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.867219 29278 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:20:45.867657 29176 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.868227 29045 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:45.869808 29315 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:20:45.870085 29176 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:45.874934 29218 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.881345 29043 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.883307 29179 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:45.884281 24947 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:45.884491 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:45.885339 28899 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:45.859658 29081 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:45.886487 29081 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.888094 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2507) took 1330 ms. Trace:
W20240426 13:20:45.867483 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75341.5us
I20240426 13:20:45.888309 25617 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:20:45.890086 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 67333.2us
I20240426 13:20:45.888852 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:45.891156 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.898449 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.900547 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:45.888515 25027 rpcz_store.cc:276] 0426 13:20:44.558012 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.618719 (+ 60707us) service_pool.cc:225] Handling call
0426 13:20:45.887752 (+1269033us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b5400056ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:45.887912 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430663424}
W20240426 13:20:45.909178 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57696us
I20240426 13:20:45.909304 29281 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:45.909788 29281 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:45.909866 29293 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.910483 29218 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.910764 29281 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.911813 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.913182 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.914353 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.918228 25097 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20240426 13:20:45.918637 25097 pending_rounds.cc:76] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 9
I20240426 13:20:45.918851 25097 pending_rounds.cc:97] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20240426 13:20:45.844717 29105 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.915674 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:45.861416 29253 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:45.919119 25097 raft_consensus.cc:2866] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:45.887446 29081 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.942288 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:45.946548 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48421.1us
I20240426 13:20:45.950160 29293 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:45.950376 29009 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 13:20:45.950898 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59822.3us
I20240426 13:20:45.951169 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:45.927119 29051 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.957984 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:45.960208 29243 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:45.960391 29279 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:45.961040 28758 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:45.961637 28758 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:45.963917 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 39) took 1013 ms. Trace:
I20240426 13:20:45.964345 25101 rpcz_store.cc:276] 0426 13:20:44.950472 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.975144 (+ 24672us) service_pool.cc:225] Handling call
0426 13:20:45.963668 (+988524us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540009d3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:45.963814 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1467223936}
I20240426 13:20:45.966176 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2522) took 1010 ms. Trace:
I20240426 13:20:45.966710 25025 rpcz_store.cc:276] 0426 13:20:44.955374 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.955491 (+   117us) service_pool.cc:225] Handling call
0426 13:20:44.955731 (+   240us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:45.543677 (+587946us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.965819 (+422142us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54002f9c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:45.966110 (+   291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1187981696}
I20240426 13:20:45.879868 29006 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:45.986233 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:45.988364 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 731) took 1198 ms. Trace:
I20240426 13:20:45.963814 28758 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:45.936528 29016 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:45.989244 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:45.943048 28899 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:45.998435 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47357.8us
I20240426 13:20:45.956084 29218 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:20:46.001412 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64965.6us
I20240426 13:20:45.988826 24950 rpcz_store.cc:276] 0426 13:20:44.770019 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.791874 (+ 21855us) service_pool.cc:225] Handling call
0426 13:20:45.968048 (+1176174us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:45.968199 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":909422464}
I20240426 13:20:46.005028 29051 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.006712 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:46.018144 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:46.019970 29279 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:46.020524 29279 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:46.020901 29279 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:20:46.023831 29313 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20240426 13:20:46.025784 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.027084 29037 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:46.027429 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:46.028316 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.028892 29243 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:46.030323 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:46.032402 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:46.008977 28906 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.500s	user 0.010s	sys 0.003s
I20240426 13:20:46.027464 29037 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:45.540355 25099 rpcz_store.cc:276] 0426 13:20:44.456724 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.605572 (+148848us) service_pool.cc:225] Handling call
0426 13:20:44.622357 (+ 16785us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:45.050039 (+427682us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:45.050078 (+    39us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:45.184695 (+134617us) log.cc:842] Serialized 78 byte log entry
0426 13:20:45.324931 (+140236us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:45.454750 (+129819us) log.cc:842] Serialized 14 byte log entry
0426 13:20:45.512802 (+ 58052us) log.cc:842] Serialized 14 byte log entry
0426 13:20:45.512988 (+   186us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:45.539341 (+ 26353us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:45.539418 (+    77us) raft_consensus.cc:1671] finished
0426 13:20:45.539432 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:45.539619 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65830,"prepare.queue_time_us":89739,"prepare.run_cpu_time_us":145,"prepare.run_wall_time_us":4942,"raft.queue_time_us":57320,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":83200,"thread_start_us":547008,"threads_started":2,"wal-append.queue_time_us":215787}
I20240426 13:20:45.889678 29252 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.043838 25589 leader_election.cc:398] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11.
I20240426 13:20:46.047442 25589 leader_election.cc:401] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:46.032430 28870 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:46.033353 29009 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:46.060371 29094 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:46.060658 29005 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.061729 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2513) took 1502 ms. Trace:
I20240426 13:20:46.062438 29218 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11.
I20240426 13:20:46.082805 25028 rpcz_store.cc:276] 0426 13:20:44.559738 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.853742 (+294004us) service_pool.cc:225] Handling call
0426 13:20:44.853986 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:45.468147 (+614161us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.061448 (+593301us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:46.061651 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1198353280}
I20240426 13:20:46.058431 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:46.085698 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58333.5us
I20240426 13:20:46.087236 24948 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:46.064534 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.064453 29293 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:46.050240 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.066649 29252 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:46.047125 29037 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:46.067746 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.068539 24954 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:46.072154 29243 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:46.058905 29220 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.051663 29043 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.115185 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:46.079336 29009 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.124605 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:46.151275 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 119288us
I20240426 13:20:46.148401 29176 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.155103 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.076501 29253 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.105574 29094 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:46.160656 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96208.4us
W20240426 13:20:46.160821 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 160628us
I20240426 13:20:46.110006 28870 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:46.052232 28777 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:46.149452 29283 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.002s
I20240426 13:20:46.171928 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.173851 24951 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:46.151130 29009 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.150485 29016 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:46.186154 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67926us
I20240426 13:20:46.187136 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:46.169636 29148 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.546s	user 0.014s	sys 0.000s
I20240426 13:20:46.194108 25025 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20240426 13:20:46.194265 28777 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.195329 25027 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:46.194870 28777 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.194569 25025 pending_rounds.cc:76] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:20:46.202366 25025 pending_rounds.cc:97] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240426 13:20:46.195920 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.202677 25025 raft_consensus.cc:2866] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:46.197129 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:46.165778 29105 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.200556 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.213757 28777 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:46.214510 29218 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:46.214681 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.214949 29218 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:46.215595 25022 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:46.215907 29218 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.217235 29218 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:46.202239 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.218603 25105 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:46.219502 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.220124 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:46.220149 29243 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:46.220928 29243 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:46.221242 25105 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:46.222051 25105 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:46.222277 29293 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:46.223057 29043 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:46.223176 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:46.223454 29043 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:46.224197 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.224776 24948 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.224411 29043 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.225584 29043 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.114975 29279 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.231786 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 760) took 1281 ms. Trace:
I20240426 13:20:46.233824 25101 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:46.232297 25624 rpcz_store.cc:276] 0426 13:20:44.924925 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.925071 (+   146us) service_pool.cc:225] Handling call
0426 13:20:46.198006 (+1272935us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:46.206017 (+  8011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577659648}
I20240426 13:20:46.247792 24948 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
W20240426 13:20:46.251062 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72670.3us
I20240426 13:20:46.250727 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.253904 25096 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:46.252061 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.240305 25105 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.311221 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:46.292554 24948 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:46.185096 29176 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.310027 25022 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:46.284926 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.257462 29090 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:46.196513 29045 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20240426 13:20:46.257030 29008 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.354466 29008 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.356751 29008 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:46.318938 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131931us
W20240426 13:20:46.285279 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 198890us
W20240426 13:20:46.283737 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55440.6us
I20240426 13:20:46.286198 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2388) took 1331 ms. Trace:
I20240426 13:20:46.304438 29293 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.358952 24949 rpcz_store.cc:276] 0426 13:20:44.932693 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:44.942652 (+  9959us) service_pool.cc:225] Handling call
0426 13:20:44.942922 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:45.543153 (+600231us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.240275 (+697122us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54003c4bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:46.263792 (+ 23517us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1310633344}
I20240426 13:20:46.259608 29051 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.317415 29278 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.283593 29006 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:46.332551 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:46.363380 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:46.334319 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46658.1us
I20240426 13:20:46.316910 24950 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20240426 13:20:46.337080 29324 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:20:46.338240 25100 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20240426 13:20:46.337759 29094 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.338335 28870 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.339035 29148 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:46.339713 28777 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:46.340195 29243 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:46.340903 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44599.9us
I20240426 13:20:46.337713 29176 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.341836 29279 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.342849 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.379801 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:46.345307 25096 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:20:46.332509 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.359169 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.294812 25028 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20240426 13:20:46.362344 29293 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.377493 29148 consensus_queue.cc:1225] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 13.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:46.378729 24949 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:46.365923 25100 pending_rounds.cc:76] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 4
I20240426 13:20:46.388870 25100 pending_rounds.cc:97] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20240426 13:20:46.389165 25100 raft_consensus.cc:2866] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:46.384490 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:46.389863 25624 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:46.369048 24954 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:46.396320 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:46.402901 29176 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:46.394582 29090 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:46.405709 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46758.2us
I20240426 13:20:46.406486 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.406312 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 9, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 9 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:46.408064 25025 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20240426 13:20:46.408421 29090 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:46.407934 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.408911 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:46.409511 25025 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:46.410602 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.412245 25029 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:46.410535 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 7 to 8, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.415155 25026 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:46.404208 29293 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.416590 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.407703 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.418133 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.418438 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.418846 25103 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:46.420326 29252 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:46.420938 29252 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:46.420747 29148 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:46.421583 29252 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:46.423759 29320 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.255s	user 0.000s	sys 0.000s
I20240426 13:20:46.423377 29009 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.424482 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.425254 25027 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:46.382547 29006 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.426200 29006 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.427279 25618 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:46.428654 29006 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:46.429719 25022 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: [9.9-10.10]
I20240426 13:20:46.374377 29278 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.430456 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.437772 24951 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:46.436882 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:46.439488 25031 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20240426 13:20:46.419859 29323 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20240426 13:20:46.439056 25099 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:46.443280 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.443610 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:46.444139 24954 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:46.444314 25099 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:46.447153 24853 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 12. New cstate: current_term: 12 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.447708 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.449622 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.450225 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:46.446687 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:46.465617 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.453660 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.466805 25103 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:46.470953 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.453106 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.476172 25024 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:46.474123 25100 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:46.478708 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:46.480986 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.472198 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.481155 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.446341 29005 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:46.484822 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.487078 25028 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:46.481609 25101 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:46.481670 25029 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:46.495831 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:46.498706 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45559.7us
I20240426 13:20:46.499446 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.505589 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.506947 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.507228 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.507970 25022 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:46.508013 24948 raft_consensus.cc:2396] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:46.491492 29006 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:46.515915 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:46.515009 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:46.511576 29243 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:46.512874 24853 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 7 to 9, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:46.506811 29130 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.516867 29148 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.526695 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:46.530725 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.527554 25029 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:46.485191 28899 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20240426 13:20:46.518255 29006 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.524617 29017 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:46.540453 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.524152 29005 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:46.523702 28777 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.552328 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.554767 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.560915 25099 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:46.561592 25030 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:46.562572 29279 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:46.562788 29120 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.563010 29279 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.563436 29279 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:46.563366 29120 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.565464 29120 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.572541 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.526621 29243 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.541855 29066 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20240426 13:20:46.581842 29243 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.582654 29058 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:46.585829 29037 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:46.578387 25024 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20240426 13:20:46.586936 25024 pending_rounds.cc:76] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 10
I20240426 13:20:46.587193 25024 pending_rounds.cc:97] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20240426 13:20:46.587426 25024 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:46.546347 29130 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.590025 25026 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:46.586398 25022 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:46.545415 29006 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.591976 25099 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20240426 13:20:46.550040 29017 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:46.563081 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 289) took 30845 ms (client timeout 15000 ms). Trace:
I20240426 13:20:46.547555 29278 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.597868 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:46.599126 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:46.600061 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49379us
I20240426 13:20:46.532069 29148 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.615911 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:46.589473 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 9, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:46.594157 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.607725 25105 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.602077 24951 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:46.616824 25028 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:46.611937 25618 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20240426 13:20:46.601091 25022 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.626474 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:46.622455 25619 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:46.631196 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.634056 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.634248 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20240426 13:20:46.635228 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.635787 24954 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.635954 24953 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:46.637101 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.638453 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:46.641745 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44828.3us
I20240426 13:20:46.645004 28758 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:46.644785 24853 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 10 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.646361 24853 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 10, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 10 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:46.571141 29218 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:46.658450 29218 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.658841 29218 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.661207 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.662474 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:46.591375 29120 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.601513 29058 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20240426 13:20:46.609341 29130 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:46.642520 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.601076 29017 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:46.647564 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.589907 29009 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:46.647532 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.649645 24948 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:46.639760 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:46.657101 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63060us
I20240426 13:20:46.693022 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.666869 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:46.670979 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 258484us
I20240426 13:20:46.642484 24954 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:46.681689 24951 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.683831 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
W20240426 13:20:46.717736 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50952.5us
I20240426 13:20:46.718616 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.670953 29293 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:46.760633 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.760780 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.764822 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.767377 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:46.674952 29179 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:46.672793 29278 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:46.761612 24954 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:46.596938 28777 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:46.597517 24852 rpcz_store.cc:269] 0426 13:20:15.727122 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:15.789384 (+ 62262us) service_pool.cc:225] Handling call
0426 13:20:46.539371 (+30749987us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:36.957116 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:37.282816 (+325700us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:37.299460 (+ 16644us) write_op.cc:427] Acquired schema lock
0426 13:20:37.299486 (+    26us) tablet.cc:660] Decoding operations
0426 13:20:37.562094 (+262608us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:37.562194 (+   100us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:37.562224 (+    30us) tablet.cc:683] Acquiring locks for 28 operations
0426 13:20:38.060617 (+498393us) tablet.cc:699] Row locks acquired
0426 13:20:38.101513 (+ 40896us) write_op.cc:260] PREPARE: finished
0426 13:20:38.189690 (+ 88177us) write_op.cc:270] Start()
0426 13:20:38.261216 (+ 71526us) write_op.cc:275] Timestamp: P: 1714137638139373 usec, L: 0
0426 13:20:38.303358 (+ 42142us) op_driver.cc:331] REPLICATION: starting
0426 13:20:38.609305 (+305947us) log.cc:842] Serialized 11110 byte log entry
0426 13:20:40.176501 (+1567196us) op_driver.cc:444] REPLICATION: finished
0426 13:20:40.804883 (+628382us) write_op.cc:300] APPLY: starting
0426 13:20:40.804982 (+    99us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:40.806774 (+  1792us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:40.857604 (+ 50830us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:41.042683 (+185079us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:41.391211 (+348528us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=56
0426 13:20:41.401285 (+ 10074us) write_op.cc:311] APPLY: finished
0426 13:20:41.406910 (+  5625us) log.cc:842] Serialized 188 byte log entry
0426 13:20:42.221244 (+814334us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:42.317816 (+ 96572us) write_op.cc:446] Released schema lock
0426 13:20:42.351922 (+ 34106us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:42.382272 (+ 30350us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:42.386390 (+  4118us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":5455,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":628048,"apply.run_cpu_time_us":51037,"apply.run_wall_time_us":1665596,"num_ops":28,"prepare.queue_time_us":483511,"prepare.run_cpu_time_us":45816,"prepare.run_wall_time_us":2865600,"raft.queue_time_us":260819,"raft.run_cpu_time_us":60504,"raft.run_wall_time_us":326032,"replication_time_us":1873018,"thread_start_us":1928790,"threads_started":5,"wal-append.queue_time_us":1634516,"wal-append.run_cpu_time_us":44546,"wal-append.run_wall_time_us":2343468}]]}
I20240426 13:20:46.769272 25022 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:20:46.684965 28758 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:46.680213 29045 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.643004 29252 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.759435 28906 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:46.799480 29278 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.805461 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:46.810213 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:46.671324 29279 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:46.786579 29030 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:46.766999 29009 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:46.766718 29037 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:46.819542 25619 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:46.815899 24953 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:46.785795 29066 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:46.786456 29243 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.833092 24948 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20240426 13:20:46.787874 29179 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:46.805907 29051 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:46.817703 28906 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.838474 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:46.807271 29293 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:46.808419 28758 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:46.841799 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:46.896183 24948 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:46.841841 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.819873 29278 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.768908 29283 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20240426 13:20:46.849962 29282 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:46.714319 29120 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:46.896195 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51152.5us
I20240426 13:20:46.800873 29220 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.036s
I20240426 13:20:46.827769 29030 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.820685 29043 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:46.869040 29051 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:46.902334 24954 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:46.909229 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:46.936142 24948 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20240426 13:20:46.868582 29179 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.895143 29009 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:46.904011 25099 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:46.932507 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 417548us
I20240426 13:20:46.902690 28758 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:46.894198 29148 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:46.937124 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 10, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 10 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:46.939317 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:46.818480 29045 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:46.906414 29005 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:46.903275 28906 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.913898 24951 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:46.945448 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:46.959290 25098 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20240426 13:20:46.967968 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50666.9us
I20240426 13:20:46.930481 29030 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:46.970522 24953 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20240426 13:20:46.982474 29333 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
W20240426 13:20:46.983565 29347 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:20:46.959803 29218 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.010015 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62640.7us
I20240426 13:20:46.941845 29051 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:46.932402 29094 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:46.906785 29243 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:46.932808 29016 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.008s	sys 0.011s
I20240426 13:20:46.942703 29043 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.045698 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:47.041687 29293 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.041025 29016 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.008s	sys 0.012s
I20240426 13:20:47.068110 24853 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:47.038743 29094 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.040764 28899 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.045956 29351 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.087648 24853 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:20:47.074753 25100 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.009s	sys 0.026s
W20240426 13:20:47.092491 25100 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.009s	sys 0.026s
I20240426 13:20:47.123623 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.072319 29252 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.076105 28777 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.084957 29094 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:47.122614 29344 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20240426 13:20:47.140957 24853 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:47.149735 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 238605us
I20240426 13:20:47.121358 29279 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.135277 29253 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.156867 24853 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 6, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:47.159621 24947 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.002s	sys 0.010s
W20240426 13:20:47.178362 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.002s	sys 0.010s
I20240426 13:20:47.178985 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2393) took 1233 ms. Trace:
I20240426 13:20:47.122171 28899 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.182231 25098 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:47.182544 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 132557us
W20240426 13:20:47.124480 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.008s	sys 0.003s
I20240426 13:20:47.186547 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.187422 24953 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
W20240426 13:20:47.186859 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.009s	sys 0.003s
I20240426 13:20:47.190600 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.191020 24954 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:47.161970 29345 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20240426 13:20:47.195354 25624 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:20:47.195664 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.196465 24948 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:47.120669 29351 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:47.204964 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 194571us
I20240426 13:20:47.178922 29120 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.183439 29016 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:47.120319 29009 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.226166 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 458842us
I20240426 13:20:47.226676 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.227388 24953 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:47.182797 29105 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:47.216879 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:47.192806 24853 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:47.179355 24947 rpcz_store.cc:276] 0426 13:20:45.945847 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:45.956321 (+ 10474us) service_pool.cc:225] Handling call
0426 13:20:46.000701 (+ 44380us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:46.004435 (+  3734us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:46.304461 (+300026us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.178683 (+874222us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:47.178731 (+    48us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:47.178771 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.178838 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.178937 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4425,"prepare.queue_time_us":407896,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":421049,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":37286,"spinlock_wait_cycles":19676032,"thread_start_us":1118529,"threads_started":3,"wal-append.queue_time_us":491725}
I20240426 13:20:47.229864 24951 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:20:47.233790 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.203850 29008 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:47.240291 29008 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:47.241271 29008 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.243041 29008 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:47.259650 25616 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:20:47.262830 25624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:47.264951 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.265656 24953 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:47.263525 25624 raft_consensus.cc:595] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:47.266450 25624 raft_consensus.cc:603] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:20:47.267994 25105 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.000s
I20240426 13:20:47.267906 24954 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:47.185899 29253 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.230159 29356 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.232997 29218 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.275398 25105 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
W20240426 13:20:47.293838 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77035.1us
I20240426 13:20:47.236994 29090 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:47.294247 25105 pending_rounds.cc:76] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 5
I20240426 13:20:47.304492 25105 pending_rounds.cc:97] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20240426 13:20:47.304803 25105 raft_consensus.cc:2866] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:47.248564 29283 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.270496 29066 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:47.309855 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2148) took 1126 ms. Trace:
I20240426 13:20:47.309648 29066 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.310209 25105 rpcz_store.cc:276] 0426 13:20:46.183402 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.220718 (+ 37316us) service_pool.cc:225] Handling call
0426 13:20:46.221045 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:46.549234 (+328189us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54001fd720. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902a558d 00007fda902931d0 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59
0426 13:20:47.309642 (+760408us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.309807 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72332160}
I20240426 13:20:47.311380 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.311532 29066 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.312228 29252 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.309414 29283 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.312970 29252 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.275589 29016 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:47.314234 29283 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.314813 29252 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.315311 29220 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:20:47.315811 29336 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
I20240426 13:20:47.316790 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 744) took 1181 ms. Trace:
I20240426 13:20:47.316944 29130 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:47.317327 29130 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:47.317139 24956 rpcz_store.cc:276] 0426 13:20:45.972384 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.004281 (+ 31897us) service_pool.cc:225] Handling call
0426 13:20:46.032206 (+ 27925us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:46.040851 (+  8645us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.040894 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:46.061912 (+ 21018us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:20:46.083259 (+ 21347us) log.cc:842] Serialized 52 byte log entry
0426 13:20:46.451316 (+368057us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:46.451392 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.451519 (+   127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.147359 (+695840us) raft_consensus.cc:1671] finished
0426 13:20:47.147385 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.154125 (+  6740us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1097,"raft.queue_time_us":2702,"raft.run_cpu_time_us":1208,"raft.run_wall_time_us":24421,"spinlock_wait_cycles":4754304,"thread_start_us":286390,"threads_started":1,"wal-append.queue_time_us":577315}
I20240426 13:20:47.318290 29130 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.298900 29120 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20240426 13:20:47.314815 29362 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20240426 13:20:47.315060 29016 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.320560 29339 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20240426 13:20:47.321736 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2147) took 1483 ms. Trace:
I20240426 13:20:47.322088 25104 rpcz_store.cc:276] 0426 13:20:45.838608 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.159687 (+321079us) service_pool.cc:225] Handling call
0426 13:20:46.176863 (+ 17176us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:46.251997 (+ 75134us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.252061 (+    64us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:46.323723 (+ 71662us) log.cc:842] Serialized 26 byte log entry
0426 13:20:46.701068 (+377345us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:46.701145 (+    77us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.701238 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.321522 (+620284us) raft_consensus.cc:1671] finished
0426 13:20:47.321545 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.321681 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":304,"raft.queue_time_us":43218,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":40544,"thread_start_us":334013,"threads_started":1,"wal-append.queue_time_us":670161}
W20240426 13:20:47.323998 29337 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20240426 13:20:47.324414 25104 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:47.322831 28777 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:47.323869 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:47.270699 29279 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.329370 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:47.303876 29343 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20240426 13:20:47.356981 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2151) took 1007 ms. Trace:
W20240426 13:20:47.312453 29355 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20240426 13:20:47.357426 25102 rpcz_store.cc:276] 0426 13:20:46.349807 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.350667 (+   860us) service_pool.cc:225] Handling call
0426 13:20:46.350963 (+   296us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:46.351287 (+   324us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.351330 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:46.351717 (+   387us) log.cc:842] Serialized 26 byte log entry
0426 13:20:46.701647 (+349930us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:46.701696 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.701776 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.356723 (+654947us) raft_consensus.cc:1671] finished
0426 13:20:47.356751 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.356902 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":909,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":87,"thread_start_us":349566,"threads_started":1,"wal-append.queue_time_us":549688}
I20240426 13:20:47.319382 29130 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:47.363631 29279 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.366209 29279 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.268167 29045 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:47.274663 29043 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.329995 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1759) took 1151 ms. Trace:
W20240426 13:20:47.378607 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 183914us
I20240426 13:20:47.314105 28906 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.341708 28899 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.334296 29066 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.344246 29252 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:47.346168 29283 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.346760 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.314049 29359 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20240426 13:20:47.349444 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:47.350155 29378 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:47.352655 29220 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:47.412539 29220 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:47.412556 29378 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:47.413406 29357 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20240426 13:20:47.413972 25025 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:47.413494 29220 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.352561 29365 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 13:20:47.353636 29361 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:20:47.354928 29105 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:47.416337 29105 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.416406 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2397) took 1392 ms. Trace:
I20240426 13:20:47.303087 29356 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.417492 29388 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:47.412784 29283 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.373667 25028 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:47.415066 29365 consensus_queue.cc:1225] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 13:20:47.354967 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1950) took 1188 ms. Trace:
I20240426 13:20:47.419294 29365 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:47.374581 25031 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:47.377456 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.413677 29378 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.419869 29365 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:47.420806 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:47.382390 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.421733 25030 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:47.421445 29365 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.341128 24853 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:47.422650 29365 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:47.416937 29105 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.370429 29243 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.425205 29105 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.419605 29283 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.415014 29220 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:47.403620 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 374098us
I20240426 13:20:47.411971 29382 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.428416 29382 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.415093 29390 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:47.417531 29379 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20240426 13:20:47.376188 29084 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.378882 25097 rpcz_store.cc:276] 0426 13:20:46.178760 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.178906 (+   146us) service_pool.cc:225] Handling call
0426 13:20:46.196256 (+ 17350us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:46.299919 (+103663us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.299965 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:46.339497 (+ 39532us) log.cc:842] Serialized 52 byte log entry
0426 13:20:46.751445 (+411948us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:47.089871 (+338426us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.091434 (+  1563us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.091807 (+   373us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.327857 (+236050us) raft_consensus.cc:1671] finished
0426 13:20:47.327876 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.328040 (+   164us) inbound_call.cc:173] Queueing success response
0426 13:20:47.362572 (+ 34532us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b4800150258. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
Metrics: {"mutex_wait_us":4120,"prepare.queue_time_us":451653,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4684,"raft.queue_time_us":48548,"raft.run_cpu_time_us":3106,"raft.run_wall_time_us":269290,"spinlock_wait_cycles":21093120,"thread_start_us":695204,"threads_started":2,"wal-append.queue_time_us":694091}
I20240426 13:20:47.443200 29084 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.444020 29351 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.444329 29030 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.445272 29084 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:47.393261 29348 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.340s	user 0.001s	sys 0.000s
I20240426 13:20:47.445514 28906 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.423120 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:20:47.446909 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 575) took 1059 ms. Trace:
I20240426 13:20:47.447275 25617 rpcz_store.cc:276] 0426 13:20:46.387024 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.387210 (+   186us) service_pool.cc:225] Handling call
0426 13:20:46.390801 (+  3591us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:46.391139 (+   338us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.391166 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:46.432490 (+ 41324us) log.cc:842] Serialized 26 byte log entry
0426 13:20:46.672464 (+239974us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:46.672537 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.672610 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.446689 (+774079us) raft_consensus.cc:1671] finished
0426 13:20:47.446711 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.446840 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3055,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":222072,"threads_started":1,"wal-append.queue_time_us":424008}
I20240426 13:20:47.347718 29179 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.450446 29179 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.452270 25100 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:47.452986 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2533) took 1018 ms. Trace:
I20240426 13:20:47.453343 25027 rpcz_store.cc:276] 0426 13:20:46.434857 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.441528 (+  6671us) service_pool.cc:225] Handling call
0426 13:20:46.466517 (+ 24989us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:46.490962 (+ 24445us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.491007 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:46.495810 (+  4803us) log.cc:842] Serialized 26 byte log entry
0426 13:20:46.914607 (+418797us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:46.914642 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.914703 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.452783 (+538080us) raft_consensus.cc:1671] finished
0426 13:20:47.452813 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.452938 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3642,"raft.queue_time_us":13728,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":16511,"thread_start_us":390603,"threads_started":1,"wal-append.queue_time_us":946196}
W20240426 13:20:47.439736 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 233996us
I20240426 13:20:47.445469 25023 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:47.416920 24950 rpcz_store.cc:276] 0426 13:20:46.024406 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.377012 (+352606us) service_pool.cc:225] Handling call
0426 13:20:46.441152 (+ 64140us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:46.456140 (+ 14988us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.456184 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:46.494762 (+ 38578us) log.cc:842] Serialized 26 byte log entry
0426 13:20:46.893209 (+398447us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:46.893251 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:46.893326 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.416153 (+522827us) raft_consensus.cc:1671] finished
0426 13:20:47.416182 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.416352 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8229,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":27662,"thread_start_us":366030,"threads_started":1,"wal-append.queue_time_us":674018}
I20240426 13:20:47.413677 29359 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:47.464432 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 723) took 1058 ms. Trace:
I20240426 13:20:47.418460 29356 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:47.470911 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.465907 25101 rpcz_store.cc:276] 0426 13:20:46.405647 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.487980 (+ 82333us) service_pool.cc:225] Handling call
0426 13:20:46.499032 (+ 11052us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:46.499232 (+   200us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:47.022161 (+522929us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.464111 (+441950us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:47.464161 (+    50us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:47.464212 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.464287 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.464385 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59155,"prepare.queue_time_us":657397,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":3013,"thread_start_us":864155,"threads_started":2,"wal-append.queue_time_us":429959}
I20240426 13:20:47.419185 24949 rpcz_store.cc:276] 0426 13:20:46.166178 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.409966 (+243788us) service_pool.cc:225] Handling call
0426 13:20:46.410380 (+   414us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:46.410834 (+   454us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:46.410866 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:46.411433 (+   567us) log.cc:842] Serialized 52 byte log entry
0426 13:20:46.507078 (+ 95645us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:46.530882 (+ 23804us) log.cc:842] Serialized 14 byte log entry
0426 13:20:46.949098 (+418216us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.230828 (+281730us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.231268 (+   440us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.354715 (+123447us) raft_consensus.cc:1671] finished
0426 13:20:47.354741 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.354906 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78167,"prepare.queue_time_us":35383,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3673,"raft.queue_time_us":517409,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":28848,"thread_start_us":708771,"threads_started":3,"wal-append.queue_time_us":375391}
I20240426 13:20:47.478274 24954 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
W20240426 13:20:47.481385 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90592.9us
I20240426 13:20:47.481653 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.481729 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.482975 24954 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:47.454903 29377 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:47.484221 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.483594 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:47.484979 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.488736 24956 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:47.489038 24948 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:47.486727 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.407735 29066 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.490648 24947 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:47.483127 24955 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:47.494362 25617 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:47.496824 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.497390 25029 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:47.498795 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.499598 24953 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:47.502123 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.503382 29383 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20240426 13:20:47.504352 29383 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:47.504731 29383 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:47.505501 29383 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.506628 29383 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:47.509863 29278 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.594s	user 0.007s	sys 0.000s
I20240426 13:20:47.510113 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:47.510351 29278 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.510928 25617 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:47.481235 29084 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.512373 29084 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.514250 29084 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:47.515305 24949 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 13:20:47.518554 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:47.519387 25029 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:47.520838 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:47.522498 24920 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:47.525823 25027 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:20:47.526209 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:47.527685 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:47.464108 29365 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20240426 13:20:47.532668 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.497814 29293 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.548312 25101 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:47.536100 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:47.536890 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.543171 25024 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:47.532388 29252 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:47.553444 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 224893us
I20240426 13:20:47.554302 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:47.552259 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51878.7us
I20240426 13:20:47.552914 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:47.559201 25097 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 13:20:47.559278 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.564464 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.529917 29130 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 13:20:47.563926 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86650.9us
I20240426 13:20:47.529289 29378 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:47.525396 29179 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.568373 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.530745 29045 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:47.585673 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.586282 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:47.587007 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.589164 25022 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:20:47.589692 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:47.565799 29006 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.955s	user 0.013s	sys 0.000s
I20240426 13:20:47.573338 29016 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.469367 29390 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.587599 25104 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:47.586383 25101 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:47.480217 29359 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.529652 29356 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.044s
I20240426 13:20:47.577050 29130 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.576071 29378 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.567112 29105 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.562263 28777 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:47.597633 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:47.605311 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.609828 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:47.645069 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 85873.6us
I20240426 13:20:47.606415 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:47.581581 29253 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:47.609082 25097 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:47.650055 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.559047 29293 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.645355 25102 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:47.651906 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.501005 29377 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.651692 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 98854.6us
I20240426 13:20:47.607692 29043 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:47.674321 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.701601 25097 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:47.662223 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 724) took 1256 ms. Trace:
I20240426 13:20:47.663364 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.666000 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.665685 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.703696 25099 rpcz_store.cc:276] 0426 13:20:46.405877 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.959205 (+553328us) service_pool.cc:225] Handling call
0426 13:20:47.015421 (+ 56216us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:47.015678 (+   257us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:47.218155 (+202477us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.569106 (+350951us) log.cc:842] Serialized 14 byte log entry
0426 13:20:47.661931 (+ 92825us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:47.661980 (+    49us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:47.662024 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.662079 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.662175 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113767,"prepare.queue_time_us":216618,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":2505,"raft.queue_time_us":93149,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":524350,"threads_started":2,"wal-append.queue_time_us":439295}
I20240426 13:20:47.608136 29378 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:47.708516 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.601819 29278 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:47.711503 29084 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:47.711863 29084 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.712208 29084 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:47.713047 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.653036 29365 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.715490 29375 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:47.715901 29375 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.716223 29375 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:47.718240 29066 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:47.720096 29283 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:47.720520 29283 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.720850 29283 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:47.606611 29130 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:47.715056 29365 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.724042 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:47.709542 25029 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:47.594779 29179 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.749455 25102 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:20:47.764760 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 193705us
W20240426 13:20:47.725119 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71875us
I20240426 13:20:47.647316 29356 consensus_queue.cc:1225] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:20:47.600596 29006 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.662128 29338 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.537s	user 0.002s	sys 0.000s
I20240426 13:20:47.647294 29252 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.665926 28906 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:20:47.664328 29401 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.661885 29366 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:47.664175 29374 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:47.724412 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.766331 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:47.767686 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:47.663761 29400 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:47.769408 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2160) took 1177 ms. Trace:
I20240426 13:20:47.770982 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.768692 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:47.791014 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66690.9us
I20240426 13:20:47.791590 24947 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:47.791919 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:47.662039 29386 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:47.703328 29220 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.077s
I20240426 13:20:47.774415 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.776221 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:47.651868 29398 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:47.707746 29359 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:47.779459 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:47.807533 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 209994us
I20240426 13:20:47.780174 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.662197 29387 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.268s
I20240426 13:20:47.602680 29016 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:47.783639 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 115955us
I20240426 13:20:47.664005 29058 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20240426 13:20:47.788797 25103 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:47.770380 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:47.764132 29008 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20240426 13:20:47.710162 29105 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:47.808805 25588 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:47.806560 24952 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:47.783757 29356 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:47.871059 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 79228.5us
I20240426 13:20:47.786532 29371 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20240426 13:20:47.872444 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 106204us
I20240426 13:20:47.936321 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:47.763515 29253 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20240426 13:20:47.793836 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:20:47.790015 29400 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.800776 29176 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:47.769711 28777 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
W20240426 13:20:47.937414 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 128704us
I20240426 13:20:47.790225 25098 rpcz_store.cc:276] 0426 13:20:46.592546 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.198608 (+606062us) service_pool.cc:225] Handling call
0426 13:20:47.218632 (+ 20024us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:47.263237 (+ 44605us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:47.263272 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:47.263618 (+   346us) log.cc:842] Serialized 26 byte log entry
0426 13:20:47.568393 (+304775us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:47.568437 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:47.568510 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:47.769144 (+200634us) raft_consensus.cc:1671] finished
0426 13:20:47.769173 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:47.769339 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52894,"raft.queue_time_us":34632,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6545,"thread_start_us":274854,"threads_started":1,"wal-append.queue_time_us":384321}
I20240426 13:20:47.802176 29398 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:47.803182 29373 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:47.873368 29120 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:47.789722 29252 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.033151 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:47.936234 29037 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:47.936350 29382 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:48.064049 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 276019us
I20240426 13:20:47.948973 29176 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.066228 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:48.046456 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
W20240426 13:20:48.070675 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 339479us
W20240426 13:20:48.073252 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 161393us
I20240426 13:20:48.011835 29400 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:48.077646 29411 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.294s	user 0.001s	sys 0.000s
I20240426 13:20:48.080264 29414 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20240426 13:20:48.063432 29364 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:20:48.082823 29403 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.423s	user 0.000s	sys 0.001s
I20240426 13:20:48.082737 28870 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:48.084784 29179 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.022022 29398 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:48.056742 29037 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.097106 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:48.107919 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 171680us
I20240426 13:20:48.115649 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:47.792467 29401 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:48.116879 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 365530us
I20240426 13:20:47.951961 29356 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.074802 29176 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:47.804188 29366 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.022513 29367 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.501s
I20240426 13:20:48.118854 29037 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:48.124215 29358 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.508s	user 0.001s	sys 0.000s
W20240426 13:20:48.124691 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58568.4us
I20240426 13:20:48.146024 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.065737 29415 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:48.153676 29415 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:48.154109 29415 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:48.154960 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:48.156365 29081 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:48.156980 29081 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:48.157877 29081 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.158564 29359 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:48.158735 29391 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.378s	user 0.001s	sys 0.000s
I20240426 13:20:48.159070 29081 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:48.159118 29359 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:47.875418 29386 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.015923 29369 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.515s
W20240426 13:20:48.138393 29353 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.511s	user 0.004s	sys 0.000s
I20240426 13:20:48.128234 29422 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.101579 29005 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:48.139729 29387 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.063920 29373 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.194859 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:48.195458 25102 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:48.121977 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.141300 24951 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:48.196720 29407 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
W20240426 13:20:48.201005 29406 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20240426 13:20:48.194635 29176 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.146998 29279 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:48.151556 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:48.151705 29410 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:48.156459 25031 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:48.160231 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.160426 29090 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 13:20:48.162340 24956 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.664s	user 0.007s	sys 0.000s
I20240426 13:20:48.089170 29120 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:48.125227 24853 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.001s	sys 0.008s
W20240426 13:20:48.221058 24853 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.002s	sys 0.009s
I20240426 13:20:48.124236 29382 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:48.089314 29045 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20240426 13:20:48.116936 29390 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.129259 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:48.130081 29414 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:48.136492 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66863.4us
I20240426 13:20:48.150087 29293 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.181298 29252 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.181515 29375 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.156569 29253 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.181922 29037 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:48.182412 29017 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.382s	user 0.011s	sys 0.000s
I20240426 13:20:48.183537 29356 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.184016 29359 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.187214 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:48.116088 29220 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.144047 24955 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:48.221807 25023 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:48.184952 29006 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.190796 29005 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:48.186282 29365 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.185336 29409 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:48.230297 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 101304us
I20240426 13:20:48.230584 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.476464 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:48.232784 29279 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:48.236909 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:48.224185 29058 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:48.238138 28758 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.221s	user 0.012s	sys 0.000s
I20240426 13:20:48.222146 28906 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:48.220644 29090 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:48.471643 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.226186 29382 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:48.474787 24953 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.975s	user 0.007s	sys 0.000s
W20240426 13:20:48.396526 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 275934us
I20240426 13:20:48.247543 28899 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:48.336773 29390 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:48.226154 24956 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:20:48.505748 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 751) took 1036 ms. Trace:
I20240426 13:20:48.506070 24956 rpcz_store.cc:276] 0426 13:20:47.469810 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.470166 (+   356us) service_pool.cc:225] Handling call
0426 13:20:48.505694 (+1035528us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:48.489145 29051 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.443s	user 0.011s	sys 0.000s
I20240426 13:20:48.239384 29410 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:48.481423 25026 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:48.240425 29120 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:48.482935 29252 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:48.489701 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:48.521020 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 290505us
W20240426 13:20:48.337816 29353 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20240426 13:20:48.195358 29387 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.494030 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.491211 29105 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.493788 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.494925 29090 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:48.482517 29009 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.011s	sys 0.019s
I20240426 13:20:48.495436 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:48.544135 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 753) took 1073 ms. Trace:
I20240426 13:20:48.544464 24953 rpcz_store.cc:276] 0426 13:20:47.470679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.479752 (+  9073us) service_pool.cc:225] Handling call
0426 13:20:48.544068 (+1064316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:48.389357 29401 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:48.353767 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 206399us
I20240426 13:20:48.550632 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:48.478521 29356 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.552361 29356 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.554251 29356 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.555337 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.479943 29220 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.558372 29220 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.559419 25029 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:20:48.475611 29016 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.482975 29058 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.189230 29422 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.560182 29058 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:48.561511 29422 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.558866 29220 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20240426 13:20:48.564342 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:48.565500 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:48.240288 29008 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:48.522207 25588 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:48.494571 29371 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:48.568789 29371 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.569307 29371 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:48.571687 29373 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.493793 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.573000 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2163) took 1928 ms. Trace:
I20240426 13:20:48.573572 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.573652 25100 rpcz_store.cc:276] 0426 13:20:46.645007 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.324973 (+679966us) service_pool.cc:225] Handling call
0426 13:20:47.325216 (+   243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:47.452769 (+127553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:48.572773 (+1120004us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540009c4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:48.572938 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291645312}
I20240426 13:20:48.575268 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:48.247680 29253 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.509660 29424 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.576352 29424 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.577512 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.578275 29424 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.499771 28899 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:48.337117 29293 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.189s
I20240426 13:20:48.579602 25024 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:48.567333 29058 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.579517 29377 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.581604 29058 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.582348 25103 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20240426 13:20:48.581702 29253 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.585909 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:48.586711 25022 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:48.545974 25027 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:48.516239 29094 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.365s	user 0.012s	sys 0.000s
I20240426 13:20:48.490173 29369 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20240426 13:20:48.487334 29279 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.513496 29365 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:48.519260 29252 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.494266 28758 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.517457 29426 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:48.538585 29120 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:48.602169 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53787.3us
W20240426 13:20:48.601320 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79169.7us
I20240426 13:20:48.595634 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.624411 25026 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:48.623713 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.353530 29414 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.609308 29293 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:48.513948 29434 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:48.547550 29009 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.011s	sys 0.019s
I20240426 13:20:48.513227 29425 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:48.518528 29390 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:48.611091 25622 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:48.629884 29439 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.639008 29422 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:48.644094 25105 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:48.482748 29409 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.654700 29293 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.611431 29094 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.649113 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:48.523293 29051 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.650606 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:48.651767 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:48.617902 29252 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:48.620528 29426 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.616338 29370 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:48.610433 29045 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:48.602890 29220 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20240426 13:20:48.675635 25617 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Deduplicated request from leader. Original: 8.9->[10.10-10.10]   Dedup: 10.10->[]
I20240426 13:20:48.609159 29379 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20240426 13:20:48.639645 29037 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:48.632391 28777 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:48.652205 29371 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.643816 29016 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.340250 29017 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.617625 29365 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:48.711889 29043 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.931s	user 0.017s	sys 0.017s
W20240426 13:20:48.773953 29043 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.017s	sys 0.018s
I20240426 13:20:48.774199 28900 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:48.659238 29176 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.670984 29425 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:48.614884 29382 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.747532 29045 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:48.722205 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146682us
I20240426 13:20:48.813105 25618 raft_consensus.cc:2354] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
W20240426 13:20:48.694316 29084 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.982s	user 0.008s	sys 0.002s
I20240426 13:20:48.727990 25619 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:48.636348 29120 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.680744 29008 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:48.691601 29373 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:48.731110 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 543941us
I20240426 13:20:48.734561 25030 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:48.714593 29293 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:48.737834 25096 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:48.739670 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.739636 25617 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20240426 13:20:48.715299 29422 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:48.644820 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 96255.5us
I20240426 13:20:48.726711 29369 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:48.718525 29426 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:48.726986 29375 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:48.750497 29130 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.972s	user 0.010s	sys 0.000s
I20240426 13:20:48.774273 29043 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:48.775432 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.665810 29366 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:48.911103 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.780836 29016 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.751912 29416 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:48.718664 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 71556.8us
I20240426 13:20:48.730948 29058 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20240426 13:20:48.831123 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:48.775704 29402 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.760735 29386 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.774914 29409 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:48.811578 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 244165us
I20240426 13:20:48.837795 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:48.691076 28906 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.689908 29434 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.752331 29218 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.063s
I20240426 13:20:48.841506 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.844355 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:48.854763 29030 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.855834 29008 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.733572 29370 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.861107 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:48.965220 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 104205us
I20240426 13:20:48.721771 29410 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.864216 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:48.864329 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 125979us
I20240426 13:20:48.783813 29425 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:48.948266 29414 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.887274 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2551) took 1391 ms. Trace:
I20240426 13:20:48.775646 28777 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.734185 29441 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:48.895303 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 803) took 1978 ms (client timeout 2448 ms). Trace:
W20240426 13:20:48.916903 25102 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.000s
I20240426 13:20:48.838726 29442 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
W20240426 13:20:48.906759 29383 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.400s	user 0.008s	sys 0.007s
I20240426 13:20:48.877104 29387 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.785248 28900 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.948325 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:48.951499 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 113778us
W20240426 13:20:48.925536 29278 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.008s	sys 0.019s
I20240426 13:20:48.887954 28758 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.902135 29279 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:48.838816 29084 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:48.965732 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:48.690274 29439 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:48.968787 29449 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:48.814337 29390 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:48.977308 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 113210us
I20240426 13:20:48.865334 29373 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:48.881342 29379 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20240426 13:20:48.775017 29037 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20240426 13:20:48.979910 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.958748 29386 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:48.973443 29283 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.252s	user 0.012s	sys 0.000s
I20240426 13:20:49.010533 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:49.012526 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:48.982575 25030 rpcz_store.cc:276] 0426 13:20:47.495761 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.495880 (+   119us) service_pool.cc:225] Handling call
0426 13:20:47.529732 (+ 33852us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.837198 (+1307466us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:48.886959 (+ 49761us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:48.887204 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2770205056}
I20240426 13:20:48.918439 29130 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:49.018748 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:49.019490 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:49.016191 25102 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
W20240426 13:20:49.015411 25618 rpcz_store.cc:269] 0426 13:20:46.916755 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:46.916930 (+   175us) service_pool.cc:225] Handling call
0426 13:20:48.895056 (+1978126us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:48.895235 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4128010496}
I20240426 13:20:48.974380 29253 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:48.982250 29066 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.022s	sys 0.026s
W20240426 13:20:49.069746 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 121529us
I20240426 13:20:48.981904 29409 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:49.071338 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:48.961473 29366 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.060s
W20240426 13:20:49.029464 29179 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.016s	sys 0.006s
I20240426 13:20:48.983415 29094 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:49.063171 29278 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.008s	sys 0.019s
W20240426 13:20:49.080513 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100689us
I20240426 13:20:49.021828 29051 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:49.022903 29442 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:49.090432 25104 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.006s	sys 0.025s
I20240426 13:20:48.982132 29423 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.364s
I20240426 13:20:49.075762 29451 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:49.072153 29414 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:49.086370 29105 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20240426 13:20:49.081794 29449 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:49.085086 29017 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:48.983821 29045 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:49.093341 24949 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:49.101042 24920 leader_election.cc:398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:48.949120 29416 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:49.099784 24952 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:48.953296 29402 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:49.078195 28758 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:49.106812 25097 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.871s	user 0.010s	sys 0.012s
W20240426 13:20:49.107193 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 267360us
I20240426 13:20:49.110812 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:49.091929 25100 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:48.981943 29422 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:49.068064 29441 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:49.099658 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 133986us
I20240426 13:20:49.078982 29043 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:49.570183 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 811) took 1084 ms. Trace:
W20240426 13:20:49.727355 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.003s	sys 0.008s
W20240426 13:20:49.908655 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.007s	user 0.006s	sys 0.008s
I20240426 13:20:49.094102 29386 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:49.107746 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:49.293114 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 808) took 1281 ms. Trace:
W20240426 13:20:49.906904 25104 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.424s	user 0.034s	sys 0.052s
I20240426 13:20:49.676265 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:49.130123 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58867.6us
I20240426 13:20:49.787657 25022 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:20:49.242277 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:49.290812 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 727) took 2019 ms. Trace:
I20240426 13:20:49.900624 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:49.672394 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:49.912204 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:49.672940 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:49.142004 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 145499us
W20240426 13:20:49.916421 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 815462us
W20240426 13:20:49.915949 25097 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.680s	user 0.019s	sys 0.047s
I20240426 13:20:49.996807 24952 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:49.996598 25027 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:49.997717 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 765) took 1018 ms. Trace:
I20240426 13:20:49.917024 25616 rpcz_store.cc:276] 0426 13:20:48.485523 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.485658 (+   135us) service_pool.cc:225] Handling call
0426 13:20:48.489809 (+  4151us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:48.489992 (+   183us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:48.811846 (+321854us) log.cc:842] Serialized 14 byte log entry
0426 13:20:49.569781 (+757935us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:49.569839 (+    58us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:49.569892 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:49.569973 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:49.570112 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":487282,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":816689,"threads_started":3}
I20240426 13:20:49.908843 25102 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:49.998215 24952 rpcz_store.cc:276] 0426 13:20:48.979050 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.979433 (+   383us) service_pool.cc:225] Handling call
0426 13:20:49.997626 (+1018193us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:49.288355 29450 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20240426 13:20:49.244135 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:50.006987 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94854.2us
I20240426 13:20:50.006991 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 735) took 1037 ms. Trace:
W20240426 13:20:49.242439 29398 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.111s	user 0.010s	sys 0.000s
I20240426 13:20:50.010294 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:50.009250 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:49.118770 29179 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.016s	sys 0.006s
I20240426 13:20:50.010888 25619 rpcz_store.cc:276] 0426 13:20:48.011559 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.012706 (+  1147us) service_pool.cc:225] Handling call
0426 13:20:49.292922 (+1280216us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:49.293048 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1588350336}
I20240426 13:20:50.007275 25024 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:50.010367 25025 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240426 13:20:50.011740 25028 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:50.011718 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:50.001896 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 762) took 1367 ms. Trace:
I20240426 13:20:49.300923 29420 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:49.998421 29386 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:49.999827 29356 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:49.104173 29283 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:49.111652 29379 consensus_queue.cc:1225] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:20:49.143028 29401 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:49.093261 29371 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:49.288223 29455 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:49.142521 29417 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:49.267807 29429 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 1.001s
I20240426 13:20:49.267453 29446 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 1.006s
I20240426 13:20:49.922694 29451 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:49.266924 29399 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:49.168831 29365 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:49.300498 29445 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:49.991714 29081 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.832s	user 0.004s	sys 0.008s
W20240426 13:20:49.993296 29461 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.690s	user 0.001s	sys 0.000s
W20240426 13:20:49.923004 29400 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.813s	user 0.012s	sys 0.000s
W20240426 13:20:49.990240 29415 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.836s	user 0.012s	sys 0.000s
W20240426 13:20:49.992705 29460 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.285s	user 0.000s	sys 0.000s
W20240426 13:20:49.993917 29471 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.283s	user 0.000s	sys 0.000s
W20240426 13:20:49.990933 29463 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.748s	user 0.000s	sys 0.001s
I20240426 13:20:49.285070 29413 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:49.998818 25105 rpcz_store.cc:276] 0426 13:20:47.271707 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.311173 (+ 39466us) service_pool.cc:225] Handling call
0426 13:20:49.290574 (+1979401us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003ee9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:49.290717 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3063643904}
W20240426 13:20:49.107297 29066 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.022s	sys 0.027s
I20240426 13:20:50.027462 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:49.902204 29084 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.041379 24947 rpcz_store.cc:276] 0426 13:20:48.634257 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.634396 (+   139us) service_pool.cc:225] Handling call
0426 13:20:48.652432 (+ 18036us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:48.652674 (+   242us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:48.901892 (+249218us) log.cc:842] Serialized 14 byte log entry
0426 13:20:50.001506 (+1099614us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:50.001571 (+    65us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:50.001632 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:50.001712 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.001838 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95106,"prepare.queue_time_us":162295,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":1231352,"threads_started":2}
I20240426 13:20:49.244740 29279 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:49.651466 29452 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.581s	user 0.000s	sys 0.001s
I20240426 13:20:49.582329 29383 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.212144 29383 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.213900 29451 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.215883 29383 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:49.912854 28777 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.017688 25102 rpcz_store.cc:276] 0426 13:20:48.969497 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:49.110520 (+141023us) service_pool.cc:225] Handling call
0426 13:20:50.006924 (+896404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.216926 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2184) took 1736 ms. Trace:
I20240426 13:20:50.217455 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.218612 25102 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.217377 25103 rpcz_store.cc:276] 0426 13:20:48.480975 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.587711 (+106736us) service_pool.cc:225] Handling call
0426 13:20:48.587966 (+   255us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.681992 (+ 94026us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.682019 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:48.682395 (+   376us) log.cc:842] Serialized 26 byte log entry
0426 13:20:48.860359 (+177964us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18004db4e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:49.230211 (+369852us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:49.230265 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:49.230360 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.216603 (+986243us) raft_consensus.cc:1671] finished
0426 13:20:50.216628 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.216863 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92683,"raft.queue_time_us":123342,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":14525,"spinlock_wait_cycles":25547008,"thread_start_us":369601,"threads_started":1,"wal-append.queue_time_us":1422991}
I20240426 13:20:50.049746 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.050595 29402 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.050732 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.220532 25028 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:50.051688 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:49.999812 29447 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240426 13:20:49.284683 29428 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 1.074s
I20240426 13:20:49.284828 29376 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 1.075s
I20240426 13:20:49.999645 29008 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:49.267938 29453 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:49.284969 29427 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:49.910007 29373 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.025862 29416 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:50.028267 29444 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
I20240426 13:20:50.026441 28870 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:50.023128 29441 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.019735 29017 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.228217 29402 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:49.989987 29430 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.091s
I20240426 13:20:50.082655 29439 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.086869 29253 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:50.088814 29351 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.089905 29449 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.090426 29442 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:50.074929 29464 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.195s	user 0.000s	sys 0.004s
W20240426 13:20:50.093994 29090 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.501s	user 0.009s	sys 0.001s
I20240426 13:20:50.094956 29479 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:50.095661 29005 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.613s	user 0.006s	sys 0.004s
I20240426 13:20:49.999413 29051 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:49.892069 28900 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:50.097365 29378 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 2.349s	user 0.006s	sys 0.005s
I20240426 13:20:49.653693 29016 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:49.583009 29359 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:49.143245 29409 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.048091 29398 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:49.912645 28758 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:49.222764 29120 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.027808 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:49.267311 29462 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:50.305197 29476 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20240426 13:20:50.305076 29462 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:50.306180 29467 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20240426 13:20:50.307446 29462 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.236083 29453 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:50.013736 29433 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.036s	user 0.014s	sys 0.008s
W20240426 13:20:50.308213 29433 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.335s	user 0.014s	sys 0.009s
I20240426 13:20:49.267614 29434 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.014833 29130 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:49.922529 29414 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.309331 29453 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.108064 29481 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:50.104168 29356 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.134845 29365 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.203164 29485 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:50.207787 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2564) took 1723 ms. Trace:
I20240426 13:20:50.165496 29413 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:50.140830 29400 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.205643 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.206250 29279 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.211452 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2181) took 1989 ms. Trace:
I20240426 13:20:50.329794 25101 rpcz_store.cc:276] 0426 13:20:48.222518 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.483447 (+260929us) service_pool.cc:225] Handling call
0426 13:20:48.502641 (+ 19194us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.507861 (+  5220us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.507893 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:48.516569 (+  8676us) log.cc:842] Serialized 26 byte log entry
0426 13:20:48.782737 (+266168us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:48.783995 (+  1258us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:48.784117 (+   122us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.211223 (+1427106us) raft_consensus.cc:1671] finished
0426 13:20:50.211249 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.211389 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":187917,"raft.queue_time_us":6571,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":12261,"thread_start_us":152698,"threads_started":1,"wal-append.queue_time_us":417987}
I20240426 13:20:50.219116 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1770) took 1149 ms. Trace:
W20240426 13:20:50.331712 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 486654us
I20240426 13:20:50.331799 25102 rpcz_store.cc:276] 0426 13:20:49.069735 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.217151 (+1147416us) service_pool.cc:225] Handling call
0426 13:20:50.219055 (+  1904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.221375 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:50.221544 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 169962us
I20240426 13:20:50.035518 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.135618 29415 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:50.335311 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 326143us
I20240426 13:20:50.115613 29417 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:50.225374 29252 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.495s	user 0.011s	sys 0.000s
W20240426 13:20:50.248545 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.14082e+06us
I20240426 13:20:50.077359 29176 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:50.318670 29481 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:50.261173 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 238891us
W20240426 13:20:50.250473 29293 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.393s	user 0.010s	sys 0.001s
I20240426 13:20:50.265640 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.277043 29005 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.267081 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.268831 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20240426 13:20:50.235767 29008 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.273869 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2176) took 2060 ms. Trace:
I20240426 13:20:50.275547 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.277935 25030 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:50.264648 29425 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.197s	user 0.004s	sys 0.006s
I20240426 13:20:50.238211 29427 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.260470 29402 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:50.278419 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.973s	user 0.052s	sys 0.081s
I20240426 13:20:50.296844 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.100010 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2568) took 1367 ms. Trace:
I20240426 13:20:50.277550 29442 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:50.273486 29366 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:50.255689 29017 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.278501 28777 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:50.269491 29351 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:50.335084 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1771) took 1260 ms (client timeout 1641 ms). Trace:
I20240426 13:20:50.278415 29479 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.302502 29359 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.311568 28758 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:50.274729 29090 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.324016 25031 rpcz_store.cc:276] 0426 13:20:48.484030 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.530536 (+ 46506us) service_pool.cc:225] Handling call
0426 13:20:48.600939 (+ 70403us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.618359 (+ 17420us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.618406 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:48.618921 (+   515us) log.cc:842] Serialized 26 byte log entry
0426 13:20:48.985609 (+366688us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:48.985668 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:48.985762 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.207523 (+1221761us) raft_consensus.cc:1671] finished
0426 13:20:50.207558 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.207723 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16101,"raft.queue_time_us":200576,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":25683,"thread_start_us":290515,"threads_started":1,"wal-append.queue_time_us":569827}
I20240426 13:20:50.338129 25103 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.338680 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.228699 29283 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.342200 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2179) took 2120 ms. Trace:
I20240426 13:20:50.248440 28870 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:50.296206 29378 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.230299 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 581) took 1405 ms. Trace:
I20240426 13:20:50.364851 24947 pending_rounds.cc:76] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
W20240426 13:20:50.474879 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133971us
I20240426 13:20:50.326963 29413 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:50.646764 24952 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:50.496244 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2168) took 2959 ms (client timeout 3114 ms). Trace:
W20240426 13:20:50.617884 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2186) took 2080 ms (client timeout 1816 ms). Trace:
W20240426 13:20:50.650352 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2191) took 1804 ms (client timeout 2222 ms). Trace:
W20240426 13:20:50.650970 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.516s	user 0.059s	sys 0.083s
I20240426 13:20:50.430239 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:50.651332 25102 rpcz_store.cc:269] 0426 13:20:48.845916 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.338403 (+1492487us) service_pool.cc:225] Handling call
0426 13:20:50.650298 (+311895us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.652909 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:50.391364 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:50.429123 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 64308.9us
I20240426 13:20:50.474606 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:50.479810 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.032s	user 0.004s	sys 0.005s
I20240426 13:20:50.654242 25619 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:20:50.429847 24953 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:50.647171 24947 pending_rounds.cc:97] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20240426 13:20:50.430503 24948 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:50.677913 24954 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20240426 13:20:50.673602 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1777) took 1442 ms (client timeout 1791 ms). Trace:
W20240426 13:20:50.563163 29489 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20240426 13:20:50.567013 25056 heartbeater.cc:644] Failed to heartbeat to 127.23.61.62:34153 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.61.62:34153 timed out after 15.000s (SENT)
W20240426 13:20:50.679230 25105 rpcz_store.cc:269] 0426 13:20:49.074405 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.205302 (+1130897us) service_pool.cc:225] Handling call
0426 13:20:50.335018 (+129716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.678535 25097 rpcz_store.cc:276] 0426 13:20:48.213714 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.213833 (+   119us) service_pool.cc:225] Handling call
0426 13:20:48.214196 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:48.214456 (+   260us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:50.007450 (+1792994us) log.cc:842] Serialized 14 byte log entry
0426 13:20:50.273197 (+265747us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:50.273265 (+    68us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:50.273327 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:50.273639 (+   312us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.273785 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25366,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":1525,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6490,"thread_start_us":2036636,"threads_started":2,"wal-append.queue_time_us":488679}
W20240426 13:20:50.682368 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.451s	user 0.008s	sys 0.009s
W20240426 13:20:50.650583 25096 rpcz_store.cc:269] 0426 13:20:47.521921 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.522048 (+   127us) service_pool.cc:225] Handling call
0426 13:20:50.481423 (+2959375us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400384fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:50.481582 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2732439808}
I20240426 13:20:50.649160 25026 rpcz_store.cc:276] 0426 13:20:48.732694 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.732850 (+   156us) service_pool.cc:225] Handling call
0426 13:20:48.754418 (+ 21568us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.778241 (+ 23823us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.778291 (+    50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:48.827150 (+ 48859us) log.cc:842] Serialized 26 byte log entry
0426 13:20:49.173168 (+346018us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:49.173228 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:49.173329 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.099812 (+926483us) raft_consensus.cc:1671] finished
0426 13:20:50.099834 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.099956 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20298,"raft.queue_time_us":155813,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8226,"thread_start_us":274205,"threads_started":1,"wal-append.queue_time_us":1259908}
I20240426 13:20:50.759186 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:50.341260 29176 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:50.760627 24952 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:50.760368 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74951.1us
W20240426 13:20:50.760756 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 461580us
I20240426 13:20:50.759171 25622 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:50.563181 29409 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.366722 29481 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:50.763527 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.693544 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:50.681026 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 733) took 1850 ms (client timeout 2213 ms). Trace:
I20240426 13:20:50.648074 25023 rpcz_store.cc:276] 0426 13:20:48.825153 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.825299 (+   146us) service_pool.cc:225] Handling call
0426 13:20:48.868062 (+ 42763us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:48.884756 (+ 16694us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.884799 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:48.907581 (+ 22782us) log.cc:842] Serialized 52 byte log entry
0426 13:20:49.070473 (+162892us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b180043b700. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:49.237223 (+166750us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:49.237276 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:49.237365 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.227667 (+990302us) raft_consensus.cc:1671] finished
0426 13:20:50.227695 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.230229 (+  2534us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99772,"raft.queue_time_us":96416,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":6170,"spinlock_wait_cycles":15198080,"thread_start_us":166543,"threads_started":1,"wal-append.queue_time_us":1083643}
I20240426 13:20:50.776114 24956 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:50.809873 25101 rpcz_store.cc:269] 0426 13:20:48.831135 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.340516 (+1509381us) service_pool.cc:225] Handling call
0426 13:20:50.680953 (+340437us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:50.492314 29426 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.573s	user 0.011s	sys 0.000s
I20240426 13:20:50.583875 29370 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.563503 29356 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.621636 29379 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
W20240426 13:20:50.803400 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 2.213s (SENT) [suppressed 8 similar messages]
W20240426 13:20:50.860972 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57601.6us
I20240426 13:20:50.582754 29398 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:50.612601 29473 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.444s	user 0.000s	sys 0.001s
W20240426 13:20:50.867516 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 528375us
I20240426 13:20:50.801007 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.683562 24947 raft_consensus.cc:2866] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:50.689461 25103 rpcz_store.cc:269] 0426 13:20:49.231026 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.219819 (+988793us) service_pool.cc:225] Handling call
0426 13:20:50.673524 (+453705us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.682245 24949 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:50.763113 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:50.649622 25099 rpcz_store.cc:276] 0426 13:20:48.215673 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.215816 (+   143us) service_pool.cc:225] Handling call
0426 13:20:48.239930 (+ 24114us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:48.244404 (+  4474us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.244441 (+    37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:48.474436 (+229995us) log.cc:842] Serialized 52 byte log entry
0426 13:20:48.827705 (+353269us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:48.827763 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:48.827849 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.335523 (+1507674us) raft_consensus.cc:1671] finished
0426 13:20:50.335551 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.335673 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4042,"raft.queue_time_us":4340,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":106220,"thread_start_us":299952,"threads_started":1,"wal-append.queue_time_us":740645}
I20240426 13:20:50.762838 24950 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:50.650786 25100 rpcz_store.cc:269] 0426 13:20:48.537454 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.575002 (+ 37548us) service_pool.cc:225] Handling call
0426 13:20:50.617594 (+2042592us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:50.617797 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1186065792}
W20240426 13:20:50.807715 29410 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.008s	sys 0.012s
I20240426 13:20:50.762377 24951 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:50.684906 29447 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.872077 24948 raft_consensus.cc:2354] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8.
I20240426 13:20:50.564216 29005 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.327611 29365 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:50.474831 29414 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.809047 25102 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:50.814587 28906 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.022s	sys 0.055s
I20240426 13:20:50.688928 29243 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.177s
I20240426 13:20:50.880070 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.881186 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:50.876300 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:50.881992 25615 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:50.878099 29447 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.672892 28870 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:50.684706 29425 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.885119 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.885450 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.885685 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.886288 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:20:50.848800 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.886554 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:50.762591 29367 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20240426 13:20:50.887758 25619 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:50.656672 29279 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.808929 29400 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.656841 29009 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.850804 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.568436 29415 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.652953 29417 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.855650 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:50.859182 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.365473 29252 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.870657 25097 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:50.909622 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1776) took 1803 ms. Trace:
I20240426 13:20:50.909752 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.888330 29243 consensus_queue.cc:1225] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 13:20:50.856204 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:50.874451 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 225496us
I20240426 13:20:50.850962 29374 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.874715 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:50.916786 25100 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.917275 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2189) took 2102 ms. Trace:
I20240426 13:20:50.917557 25100 rpcz_store.cc:276] 0426 13:20:48.814500 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.874478 (+2059978us) service_pool.cc:225] Handling call
0426 13:20:50.917224 (+ 42746us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.391836 29293 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.876952 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:50.877043 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.877786 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.688227 28777 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.878453 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.921351 25623 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:20:50.921590 25103 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:50.872341 29462 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:50.889222 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:50.871467 29410 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.008s	sys 0.012s
I20240426 13:20:50.889997 25023 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:50.871424 29501 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20240426 13:20:50.890326 25031 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:50.890825 25099 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.760231 29253 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.089s
I20240426 13:20:50.860610 29379 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:50.874989 28906 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.033s	sys 0.078s
I20240426 13:20:50.874245 29278 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.896842 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.932500 25619 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:50.934630 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:50.929140 29462 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.935127 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.935542 29252 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.935765 25619 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:50.935847 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.936545 25620 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:50.936923 29008 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.936462 29379 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.937600 29379 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:50.938058 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:50.938447 25098 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.925303 29374 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.904757 29176 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.940932 29176 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:50.936548 29462 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.944453 29351 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.945741 29425 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.945875 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:50.948740 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 813) took 2319 ms. Trace:
I20240426 13:20:50.899473 24951 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:50.900641 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:50.886924 25101 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:50.856784 29426 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.949723 24947 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:50.909930 25097 rpcz_store.cc:276] 0426 13:20:49.105812 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.800769 (+1694957us) service_pool.cc:225] Handling call
0426 13:20:50.909573 (+108804us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.910830 24952 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20240426 13:20:50.873445 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1772) took 1798 ms (client timeout 2219 ms). Trace:
I20240426 13:20:50.910928 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.960902 25616 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:50.914355 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:50.961551 24954 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:50.960862 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.918368 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.962239 25097 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:50.898401 29451 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:50.922335 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:50.928848 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:50.930492 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 732) took 2106 ms (client timeout 2489 ms). Trace:
I20240426 13:20:50.953671 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:50.964895 25103 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:50.964712 25105 rpcz_store.cc:269] 0426 13:20:48.823868 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.804235 (+1980367us) service_pool.cc:225] Handling call
0426 13:20:50.930415 (+126180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.966257 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.949250 25624 rpcz_store.cc:276] 0426 13:20:48.609153 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.617497 (+  8344us) service_pool.cc:225] Handling call
0426 13:20:48.619756 (+  2259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:48.620101 (+   345us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:48.620139 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:48.635689 (+ 15550us) log.cc:842] Serialized 26 byte log entry
0426 13:20:48.881560 (+245871us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b1800337060. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:49.115445 (+233885us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:49.115497 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:49.115591 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:50.921364 (+1805773us) raft_consensus.cc:1671] finished
0426 13:20:50.921387 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.928955 (+  7568us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":352,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":9134,"spinlock_wait_cycles":2490752,"thread_start_us":216075,"threads_started":1,"wal-append.queue_time_us":1463659}
I20240426 13:20:50.758816 29366 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:50.963016 25102 rpcz_store.cc:269] 0426 13:20:49.074723 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.693259 (+1618536us) service_pool.cc:225] Handling call
0426 13:20:50.873368 (+180109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.963356 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:50.962407 25100 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:50.968669 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.969225 25100 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:50.969957 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:50.970451 25100 raft_consensus.cc:2362] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13. Re-sending same reply.
I20240426 13:20:50.972447 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.973124 25623 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:50.973906 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:50.975841 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:50.976050 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.976756 29426 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.977384 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.978919 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.979681 25105 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:20:50.980711 29427 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.980882 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.981889 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2188) took 2167 ms. Trace:
I20240426 13:20:50.982280 25105 rpcz_store.cc:276] 0426 13:20:48.814212 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.980597 (+2166385us) service_pool.cc:225] Handling call
0426 13:20:50.981825 (+  1228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.983445 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.961815 29278 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:50.985373 28777 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.985957 29278 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:50.986884 29365 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.990134 28758 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:50.991271 25096 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:50.991863 29293 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:50.992127 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1775) took 1890 ms (client timeout 2450 ms). Trace:
W20240426 13:20:50.992481 25096 rpcz_store.cc:269] 0426 13:20:49.101545 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.762186 (+1660641us) service_pool.cc:225] Handling call
0426 13:20:50.991937 (+229751us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b540039ec54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:50.992077 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174172544}
I20240426 13:20:50.992978 29479 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:50.993688 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1774) took 1910 ms. Trace:
I20240426 13:20:50.923843 29447 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:50.993844 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 731) took 2456 ms. Trace:
I20240426 13:20:50.998039 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:50.998636 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:50.999115 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:50.994046 25103 rpcz_store.cc:276] 0426 13:20:49.083419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.922354 (+1838935us) service_pool.cc:225] Handling call
0426 13:20:50.993639 (+ 71285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:50.995072 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.001086 25097 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:51.001211 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.961198 25617 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:50.999751 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:51.003829 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.004381 24947 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:50.998271 25096 rpcz_store.cc:276] 0426 13:20:48.536977 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.993147 (+2456170us) service_pool.cc:225] Handling call
0426 13:20:50.993416 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:50.993572 (+   156us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:50.993595 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:50.993633 (+    38us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:20:50.993660 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:50.993712 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:50.993800 (+    88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.003800 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:50.999786 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2193) took 1006 ms. Trace:
I20240426 13:20:51.005770 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.006121 25101 rpcz_store.cc:276] 0426 13:20:49.993157 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.951211 (+958054us) service_pool.cc:225] Handling call
0426 13:20:50.999733 (+ 48522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.006553 25103 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:50.907557 29009 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.002102 25029 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:50.916435 29402 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.008216 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.089046 24948 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20240426 13:20:51.050079 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.007846 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:51.089601 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 91658.3us
I20240426 13:20:51.090116 24952 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:51.090135 24947 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:51.007184 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 734) took 2090 ms. Trace:
I20240426 13:20:51.095254 25102 rpcz_store.cc:276] 0426 13:20:48.917954 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.006632 (+2088678us) service_pool.cc:225] Handling call
0426 13:20:51.006854 (+   222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:51.006987 (+   133us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:51.007004 (+    17us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:51.007030 (+    26us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:51.007054 (+    24us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.007086 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:51.007149 (+    63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.071813 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:51.069927 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:51.049980 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96395.3us
I20240426 13:20:51.102032 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.103076 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.106976 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.107595 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:51.109731 25101 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [10.9-12.10]
I20240426 13:20:51.088402 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2192) took 1095 ms. Trace:
I20240426 13:20:51.111239 25100 rpcz_store.cc:276] 0426 13:20:49.992812 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.087623 (+1094811us) service_pool.cc:225] Handling call
0426 13:20:51.087882 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:51.088060 (+   178us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:51.088085 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:51.088129 (+    44us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:51.088192 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.088249 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:51.088347 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.088402 25097 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:51.117472 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 736) took 1974 ms. Trace:
I20240426 13:20:51.117851 25097 rpcz_store.cc:276] 0426 13:20:49.142623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.994842 (+1852219us) service_pool.cc:225] Handling call
0426 13:20:51.117403 (+122561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.090546 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:51.005461 29386 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.901s	user 0.009s	sys 0.002s
I20240426 13:20:51.140311 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.139734 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1773) took 2064 ms. Trace:
I20240426 13:20:51.018397 29410 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:51.149714 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59234.2us
I20240426 13:20:51.140981 25101 rpcz_store.cc:276] 0426 13:20:49.075006 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.006739 (+1931733us) service_pool.cc:225] Handling call
0426 13:20:51.100766 (+ 94027us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:51.139477 (+ 38711us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.139683 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.078176 29359 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.079242 29447 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:51.017102 29090 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.100719 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2194) took 1107 ms. Trace:
I20240426 13:20:51.160404 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.063464 29293 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:51.147132 29500 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:51.140702 29418 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:51.168138 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.170078 25102 rpcz_store.cc:276] 0426 13:20:49.993500 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.099109 (+1105609us) service_pool.cc:225] Handling call
0426 13:20:51.100251 (+  1142us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:51.100460 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:51.100494 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:51.100530 (+    36us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:51.100556 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.100597 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:51.100674 (+    77us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:51.170914 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.154467 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.165683 25618 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:51.178264 28899 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.182758 29382 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.183913 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1803) took 1941 ms. Trace:
I20240426 13:20:51.184181 29447 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:51.202535 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48055us
I20240426 13:20:51.008752 29440 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:51.188779 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48553.6us
I20240426 13:20:51.063683 29414 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:50.997278 29351 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20240426 13:20:50.997363 29451 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.148452 29446 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
W20240426 13:20:51.191429 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 305322us
I20240426 13:20:51.166141 25100 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:51.199851 25030 rpcz_store.cc:276] 0426 13:20:49.242201 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:49.242317 (+   116us) service_pool.cc:225] Handling call
0426 13:20:51.183710 (+1941393us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:51.183846 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":597980928}
I20240426 13:20:51.008358 29490 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.181s
I20240426 13:20:51.191356 29374 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:51.196568 29090 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:51.198452 24954 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:51.170339 29253 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.025141 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.203748 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.184903 29293 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:51.191872 29500 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.207966 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:51.208459 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.206382 29278 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:51.192277 29418 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.150285 29386 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.217561 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.220108 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.205902 29447 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:20:51.224560 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 199495us
I20240426 13:20:51.240852 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.207441 29440 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.244800 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:51.226501 24952 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:51.209395 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.232862 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.206259 28899 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.235319 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.211529 29414 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.218282 28758 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:51.243429 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.220984 29253 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:51.245999 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.212404 29446 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:51.246843 24948 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.247442 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.246652 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:51.225672 29293 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:51.253058 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.266083 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.247321 29408 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.233312 29500 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.270750 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.271324 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:51.258177 24949 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:51.234150 29374 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:51.234792 29278 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.261355 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:51.232676 29509 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.236135 29510 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.236138 29418 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.262274 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.294127 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.308502 25615 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:51.294512 25028 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 8.9->[10.10-10.10]   Dedup: 10.10->[]
I20240426 13:20:51.277801 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.300315 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.310194 25100 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:51.305855 25620 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Deduplicated request from leader. Original: 8.9->[10.10-10.10]   Dedup: 10.10->[]
I20240426 13:20:51.310977 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20240426 13:20:51.255735 24954 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:51.248803 29440 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.312693 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.269475 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.278340 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.269685 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.248575 29387 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:51.313956 25023 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:51.315665 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.270067 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.252290 29414 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.251958 29410 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:51.332567 25031 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:51.337451 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.272848 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.254153 29462 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:51.259492 29253 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.312119 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.353219 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.302546 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:51.302474 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.313288 25028 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20240426 13:20:51.313166 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.248456 29425 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.278015 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.277648 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:51.315974 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46354.7us
I20240426 13:20:51.356544 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.358137 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.357419 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 758) took 2713 ms. Trace:
I20240426 13:20:51.358634 25619 raft_consensus.cc:2354] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:51.359313 25025 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:51.359427 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.315433 24954 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:51.358784 24953 rpcz_store.cc:276] 0426 13:20:48.630444 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.634607 (+  4163us) service_pool.cc:225] Handling call
0426 13:20:48.634923 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:50.683228 (+2048305us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.343863 (+660635us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:51.344058 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4127485312}
I20240426 13:20:51.361325 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.327540 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2177) took 3112 ms. Trace:
W20240426 13:20:51.328195 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 988139us
I20240426 13:20:51.356209 29425 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.362722 29425 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.324524 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1278) took 3100 ms. Trace:
I20240426 13:20:51.363344 25102 rpcz_store.cc:276] 0426 13:20:48.224612 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.200710 (+2976098us) service_pool.cc:225] Handling call
0426 13:20:51.323906 (+123196us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:51.324456 (+   550us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26484480}
I20240426 13:20:51.319597 29503 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:51.364282 29503 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.366153 29503 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.277778 29278 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:51.367015 28870 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.336565 29446 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.334439 29094 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:51.369271 29094 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.329349 29009 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.336979 29430 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:51.370658 29430 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.370983 29430 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.268455 29359 raft_consensus.cc:2741] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20240426 13:20:51.362140 25104 rpcz_store.cc:276] 0426 13:20:48.214902 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.215083 (+   181us) service_pool.cc:225] Handling call
0426 13:20:48.238378 (+ 23295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:48.238580 (+   202us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:50.012649 (+1774069us) log.cc:842] Serialized 14 byte log entry
0426 13:20:50.335925 (+323276us) log.cc:842] Serialized 14 byte log entry
0426 13:20:50.943118 (+607193us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b18003106c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:51.326950 (+383832us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b18003106c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:20:51.327113 (+   163us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:51.327181 (+    68us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:51.327264 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.327347 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:51.327469 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1069824,"prepare.queue_time_us":1039185,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":127,"raft.queue_time_us":996357,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":27199,"spinlock_wait_cycles":171622400,"thread_start_us":1848181,"threads_started":2,"wal-append.queue_time_us":554304}
I20240426 13:20:51.374943 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.313021 29374 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:51.279266 29413 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.623s	user 0.006s	sys 0.011s
I20240426 13:20:51.377312 29374 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.362116 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.342664 24949 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.334131 29179 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.380416 29179 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.355777 29462 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:51.381706 29462 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.382927 29179 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:51.303541 29451 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.385448 29451 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:51.386181 29451 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:51.313632 29509 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:51.387197 29447 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.387487 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.271958 29408 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.388314 25097 raft_consensus.cc:2354] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:51.317808 29293 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.389128 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.390033 29293 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.390151 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:51.323140 29442 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.635s	user 0.005s	sys 0.011s
I20240426 13:20:51.390995 29359 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.391268 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.391481 29359 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.377473 29413 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:51.392777 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61583.7us
I20240426 13:20:51.393138 25621 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:51.393441 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.393972 29413 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.394682 25026 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.396251 29494 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:51.396518 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.334221 29510 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.398468 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.401309 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.402377 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.403098 25102 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.380973 24949 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:51.376878 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.387329 29509 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.388209 29408 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.344597 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:51.383750 29462 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.375883 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.403714 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.391857 25097 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:51.276966 29351 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.348023 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.349190 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.361644 25100 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.329911 29243 raft_consensus.cc:2741] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20240426 13:20:51.441340 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96832us
I20240426 13:20:51.324843 29386 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.350317 29387 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:51.346783 25103 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:51.449924 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.451028 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:51.462481 25024 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20240426 13:20:51.446606 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83510.5us
I20240426 13:20:51.448056 24952 raft_consensus.cc:2396] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240426 13:20:51.403309 29440 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.448810 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.470770 25025 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:51.449117 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.471997 24953 raft_consensus.cc:2396] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:51.452458 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.473052 25102 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.453449 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.474014 25097 raft_consensus.cc:2396] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20240426 13:20:51.455931 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.459110 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.460389 25028 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:20:51.449674 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:51.399904 29510 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.466915 24955 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:51.467938 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.469183 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:51.449954 29379 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.512s	user 0.000s	sys 0.018s
I20240426 13:20:51.484962 24920 leader_election.cc:398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:51.485410 24920 leader_election.cc:401] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240426 13:20:51.385310 29500 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.483687 25103 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.484030 25104 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.489473 25100 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:51.480937 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:51.493219 25026 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:51.445000 29442 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:51.497327 24956 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20240426 13:20:51.348863 29481 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.501533 25101 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20240426 13:20:51.502578 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 249311us
I20240426 13:20:51.461423 29179 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:51.513406 25097 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:51.514796 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.515022 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.515988 25097 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.517143 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.478900 29365 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.493016 28899 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.510466 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.517630 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.531060 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.532994 25104 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20240426 13:20:51.511765 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.536479 25103 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.540062 25622 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:20:51.521168 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.530755 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.545121 25101 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.545924 25100 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:20:51.507025 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 252619us
I20240426 13:20:51.548029 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:51.445822 29413 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.053s
I20240426 13:20:51.495457 29090 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:51.495389 29442 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:51.509663 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:51.558168 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48605.6us
I20240426 13:20:51.527606 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.529188 25617 raft_consensus.cc:2354] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:51.496192 29503 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:51.553236 25028 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20240426 13:20:51.554889 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63474.5us
I20240426 13:20:51.463235 29351 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.593225 24921 leader_election.cc:398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
W20240426 13:20:51.595305 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47366.1us
I20240426 13:20:51.596493 25103 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:51.598603 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.599192 24952 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.599613 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:51.511336 29510 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:51.602981 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.496744 29370 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.059s
I20240426 13:20:51.618289 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.530141 28899 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:51.623623 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.625178 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.618613 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.630998 25025 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:51.558414 29379 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.628118 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.632891 25097 raft_consensus.cc:2396] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20240426 13:20:51.637423 24949 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:51.592242 29442 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:51.594835 29090 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:51.543531 29365 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.514326 29278 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.644799 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.644944 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.598031 29388 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:51.525594 29179 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.650925 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.626133 29503 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:51.660661 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.664181 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.623894 29218 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:51.642227 29410 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:51.624545 29462 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240426 13:20:51.638423 28899 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.670913 25025 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:51.605965 29430 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.672596 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.685079 25028 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.667968 29481 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.650291 29090 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.625515 29510 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.632301 28870 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:51.630000 29009 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20240426 13:20:51.687163 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:51.695256 25097 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.663758 29388 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:51.685837 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 276752us
I20240426 13:20:51.665217 29503 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.676913 24952 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:51.664157 29451 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:51.695575 25097 raft_consensus.cc:603] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:51.644429 29379 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.688386 29494 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:51.697116 29388 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:51.717792 29176 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.751s	user 0.017s	sys 0.001s
I20240426 13:20:51.695353 29120 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:51.734009 25027 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:51.700268 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.696985 29481 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:20:51.678187 29218 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:51.697953 29503 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:51.726548 29382 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.001s
I20240426 13:20:51.722335 29440 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.735888 29500 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.737990 29481 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:51.704217 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.759320 24952 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:51.760576 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2185) took 3258 ms. Trace:
I20240426 13:20:51.760962 25105 rpcz_store.cc:276] 0426 13:20:48.501781 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.984685 (+2482904us) service_pool.cc:225] Handling call
0426 13:20:50.985074 (+   389us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:51.384749 (+399675us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:51.384777 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:51.385163 (+   386us) log.cc:842] Serialized 26 byte log entry
0426 13:20:51.632265 (+247102us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:51.632305 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.632392 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:51.760385 (+127993us) raft_consensus.cc:1671] finished
0426 13:20:51.760405 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:51.760517 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399236,"raft.queue_time_us":400227,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":246734,"threads_started":1,"wal-append.queue_time_us":363447}
I20240426 13:20:51.745118 29401 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:51.747632 29120 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:51.760629 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.782806 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:51.783607 25105 raft_consensus.cc:595] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:51.783996 25105 raft_consensus.cc:603] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:51.763288 29481 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:51.762475 29179 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:51.811894 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.830662 25029 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:51.835335 29510 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.816174 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:51.816397 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 216840us
I20240426 13:20:51.849756 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.850630 24952 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:51.845221 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:51.851132 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.857260 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.875056 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.883682 25102 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:51.876838 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:51.858422 29351 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.777065 29401 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.862375 28870 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:51.882880 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:51.867364 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:51.870980 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 247429us
I20240426 13:20:51.884800 24952 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:51.870700 28906 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:51.895234 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:51.881759 29516 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
I20240426 13:20:51.895989 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.916162 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.906416 25069 leader_election.cc:398] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8.
I20240426 13:20:51.907030 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.917336 25069 leader_election.cc:401] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240426 13:20:51.894515 25029 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:51.917667 25028 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
W20240426 13:20:51.889309 29418 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.574s	user 0.005s	sys 0.011s
I20240426 13:20:51.876502 29534 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:51.920552 24947 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:20:51.885464 29278 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.922539 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:51.923310 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.892243 28870 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:51.913298 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.924288 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:51.907756 29425 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.545s	user 0.016s	sys 0.001s
I20240426 13:20:51.929456 25104 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:51.930480 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.936710 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:51.937237 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.937562 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.938398 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.938616 25023 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:51.939218 25105 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.860468 29218 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.919710 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.946202 25100 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:51.908358 28906 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:51.889895 29430 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:51.888885 29379 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.930858 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.957474 25028 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.925114 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:51.940430 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:51.937558 24952 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.934090 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:51.920449 25102 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:20:51.899645 29410 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:51.943451 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:51.961102 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.961916 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:51.944361 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:51.950820 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 98197.3us
I20240426 13:20:51.939575 28870 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:51.922214 29440 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:51.985780 25104 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20240426 13:20:51.989917 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49579.8us
I20240426 13:20:51.931083 29009 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:51.923887 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.012s	sys 0.034s
W20240426 13:20:51.991173 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.013s	sys 0.034s
I20240426 13:20:51.976749 25623 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:51.980026 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:51.981976 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56963.6us
W20240426 13:20:51.991811 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 632405us
I20240426 13:20:51.915180 29351 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240426 13:20:52.016403 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:51.918934 29500 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20240426 13:20:51.943635 28777 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:51.937264 29366 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:51.949929 29243 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:52.017374 25623 raft_consensus.cc:595] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:52.019548 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57756.9us
I20240426 13:20:52.024448 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.020464 25023 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:52.019699 29179 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8.
I20240426 13:20:52.028488 29430 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
W20240426 13:20:52.031124 29387 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.552s	user 0.013s	sys 0.001s
I20240426 13:20:52.017350 29379 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:51.944787 29425 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.035784 29481 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:51.913661 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 14.576s	user 0.157s	sys 0.086s Metrics: {"bytes_written":30173,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":404349,"dirs.run_cpu_time_us":465,"dirs.run_wall_time_us":32172,"drs_written":1,"lbm_read_time_us":35708,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":2,"lbm_write_time_us":291355,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":6,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":104628,"threads_started":1,"update_count":28}
I20240426 13:20:51.931733 29377 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.085s
I20240426 13:20:52.037791 25028 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:52.038823 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
W20240426 13:20:52.039508 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43620.4us
I20240426 13:20:52.041800 28870 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.051896 29429 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:51.956668 29534 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:52.019747 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 124528us
I20240426 13:20:51.925763 29418 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:51.907400 29120 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.062996 24952 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
W20240426 13:20:52.082922 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89366us
I20240426 13:20:52.137593 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:52.060247 29522 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20240426 13:20:52.043694 29090 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.097059 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:51.958441 28906 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:52.091435 29414 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.741s	user 0.017s	sys 0.003s
I20240426 13:20:52.157308 29414 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.060093 29009 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:52.159621 24954 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:52.131336 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.160071 29554 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:52.160384 25028 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:52.160462 29554 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:52.161239 29554 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:52.161209 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.162384 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.163269 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.138841 25031 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12.
I20240426 13:20:52.140601 25023 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:52.166055 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.166498 25023 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:52.138856 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.085286 29387 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:52.154040 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:20:52.168778 29387 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:52.169261 29387 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.170922 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:20:52.171633 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:20:52.172502 29527 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
W20240426 13:20:52.174221 29536 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20240426 13:20:52.175097 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.155758 24952 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
W20240426 13:20:52.176640 29526 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.305s	user 0.000s	sys 0.000s
I20240426 13:20:52.177572 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.155323 29550 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:52.178033 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2190) took 3341 ms. Trace:
I20240426 13:20:52.184134 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.184808 25028 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:52.183507 25096 rpcz_store.cc:276] 0426 13:20:48.836570 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.006394 (+2169824us) service_pool.cc:225] Handling call
0426 13:20:51.052305 (+ 45911us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:51.385903 (+333598us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:51.385932 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:51.386331 (+   399us) log.cc:842] Serialized 26 byte log entry
0426 13:20:51.680691 (+294360us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:51.680749 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.680842 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:52.177804 (+496962us) raft_consensus.cc:1671] finished
0426 13:20:52.177831 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:52.177971 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":333170,"raft.queue_time_us":339598,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"thread_start_us":293993,"threads_started":1,"wal-append.queue_time_us":386607}
I20240426 13:20:52.185470 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.187772 29550 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.189831 29550 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:52.178090 24948 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:52.191002 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.192317 25028 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:52.193050 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.193514 24953 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:52.193621 25028 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:52.193640 29554 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.195283 29481 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.196357 25023 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:52.194588 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.198103 29410 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.198882 25028 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:52.202716 25028 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20240426 13:20:52.205786 25028 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [9.9-11.11]
I20240426 13:20:52.213178 28870 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:52.175659 29535 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20240426 13:20:52.136714 29534 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.219959 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8487 bytes on disk
I20240426 13:20:52.221433 29425 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.222018 29500 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:52.222573 29500 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:52.222865 29500 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:52.223193 29500 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:52.052349 29243 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.086467 29538 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:20:52.233199 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 332342us
I20240426 13:20:52.234827 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:52.141534 29523 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
I20240426 13:20:52.250734 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:52.253712 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 230082us
I20240426 13:20:52.254283 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.254559 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.256021 25589 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15.
I20240426 13:20:52.256938 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.260113 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.262653 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:52.262849 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.036s	user 0.001s	sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":439,"lbm_reads_lt_1ms":4}
W20240426 13:20:52.313625 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 174862us
I20240426 13:20:52.313455 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:52.314414 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.313985 25029 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:52.320513 24920 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:52.321573 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59527.3us
I20240426 13:20:52.260480 29366 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:52.259236 28777 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.259253 29440 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:52.326629 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:52.327628 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:52.265102 29556 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:52.285270 29538 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:52.330963 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:52.260238 28870 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20240426 13:20:52.334164 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:52.341306 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2175) took 4128 ms (client timeout 3863 ms). Trace:
I20240426 13:20:52.322572 29550 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:52.336077 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.332401 25029 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:20:52.322664 29553 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:52.350776 29558 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:52.351143 29558 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:52.351480 29558 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:52.351600 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:52.352347 29553 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:52.352499 28758 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:52.352795 29553 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:52.353190 28758 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.353686 29553 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.354305 28758 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:52.345901 25098 rpcz_store.cc:269] 0426 13:20:48.213279 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.213424 (+   145us) service_pool.cc:225] Handling call
0426 13:20:52.341117 (+4127693us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b540001ccd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.341243 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5634388736}
I20240426 13:20:52.347882 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.341178 29560 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:52.340931 29366 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:52.361886 28870 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:52.362800 28777 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.363811 29538 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.364018 29558 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.359522 29550 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.332242 29555 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:52.373195 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 590) took 1493 ms. Trace:
I20240426 13:20:52.406273 25022 rpcz_store.cc:276] 0426 13:20:50.880086 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.880274 (+   188us) service_pool.cc:225] Handling call
0426 13:20:50.880647 (+   373us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:50.880933 (+   286us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:50.880970 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:51.321314 (+440344us) spinlock_profiling.cc:243] Waited 9.97 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:20:51.321739 (+   425us) log.cc:842] Serialized 26 byte log entry
0426 13:20:51.518555 (+196816us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:51.518602 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:51.518689 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:52.358869 (+840180us) raft_consensus.cc:1671] finished
0426 13:20:52.358898 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:52.373105 (+ 14207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22940032,"thread_start_us":196446,"threads_started":1,"wal-append.queue_time_us":409442}
I20240426 13:20:52.408375 29555 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:52.409260 29558 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.409374 29243 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.410228 29550 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.411020 29402 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:52.411427 29402 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:52.412523 25097 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:52.375281 29568 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:52.384759 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 823) took 1156 ms. Trace:
I20240426 13:20:52.437192 25617 rpcz_store.cc:276] 0426 13:20:51.228320 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.245895 (+ 17575us) service_pool.cc:225] Handling call
0426 13:20:52.384489 (+1138594us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.384680 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":579721472}
W20240426 13:20:52.385684 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2417) took 4656 ms (client timeout 2342 ms). Trace:
W20240426 13:20:52.438202 24950 rpcz_store.cc:269] 0426 13:20:47.729117 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:47.729245 (+   128us) service_pool.cc:225] Handling call
0426 13:20:52.385428 (+4656183us) spinlock_profiling.cc:243] Waited 2.96 s on lock 0x7b54003c46d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.385579 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6818723712}
I20240426 13:20:52.343194 29556 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:52.448403 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1998) took 1529 ms. Trace:
W20240426 13:20:52.393796 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 280452us
I20240426 13:20:52.342386 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:52.450966 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 108710us
I20240426 13:20:52.391635 29366 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:52.440057 24920 leader_election.cc:398] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:52.453317 24920 leader_election.cc:401] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:52.330940 29351 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:52.405308 25098 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20240426 13:20:52.403378 29408 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.953s	user 0.014s	sys 0.004s
I20240426 13:20:52.448940 24948 rpcz_store.cc:276] 0426 13:20:50.918518 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.271899 (+353381us) service_pool.cc:225] Handling call
0426 13:20:52.448180 (+1176281us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b54000a69d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.448315 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1875941760}
I20240426 13:20:52.454635 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.460741 29366 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:52.456425 29555 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:52.480829 25022 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240426 13:20:52.481267 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1820) took 1210 ms. Trace:
I20240426 13:20:52.486670 25026 rpcz_store.cc:276] 0426 13:20:51.270755 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.337734 (+ 66979us) service_pool.cc:225] Handling call
0426 13:20:52.481124 (+1143390us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b54002fa154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.481218 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35612928}
W20240426 13:20:52.481575 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1991) took 1572 ms (client timeout 1663 ms). Trace:
W20240426 13:20:52.489213 24956 rpcz_store.cc:269] 0426 13:20:50.908813 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.945641 (+ 36828us) service_pool.cc:225] Handling call
0426 13:20:52.481422 (+1535781us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.481526 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1266106624}
I20240426 13:20:52.483556 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:52.469633 29568 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:52.485824 29413 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.012s	sys 0.047s
W20240426 13:20:52.499287 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 171723us
W20240426 13:20:52.500878 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 164866us
I20240426 13:20:52.501335 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.501804 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.502987 24956 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:20:52.482761 29566 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:52.505692 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:52.493844 29503 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.749s	user 0.010s	sys 0.000s
I20240426 13:20:52.466631 29408 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.510623 25620 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:20:52.511111 25022 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:20:52.511340 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:52.505558 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:52.503465 29413 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.012s	sys 0.047s
I20240426 13:20:52.505621 25623 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:52.477262 29009 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.022s
I20240426 13:20:52.488940 29563 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:52.533392 29563 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.533596 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 755) took 3906 ms. Trace:
I20240426 13:20:52.533988 24955 rpcz_store.cc:276] 0426 13:20:48.627702 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.627840 (+   138us) service_pool.cc:225] Handling call
0426 13:20:48.649144 (+ 21304us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:51.478738 (+2829594us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.533375 (+1054637us) spinlock_profiling.cc:243] Waited 2.81 s on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:52.533545 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6467804544}
I20240426 13:20:52.534308 29563 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.535279 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:52.509506 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1997) took 1591 ms. Trace:
I20240426 13:20:52.536327 24956 rpcz_store.cc:276] 0426 13:20:50.918281 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.490247 (+1571966us) service_pool.cc:225] Handling call
0426 13:20:52.502797 (+ 12550us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:52.509274 (+  6477us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.509457 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:52.536739 29534 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.537310 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:52.538144 24956 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:52.538874 29008 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:52.539021 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.539347 29008 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:52.539749 29008 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:52.492431 29401 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.541339 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:52.542809 29509 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.097s	user 0.017s	sys 0.000s
I20240426 13:20:52.542873 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.543341 29509 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.544638 29481 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:52.535876 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:52.540123 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.547428 24956 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:52.550356 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 44848.3us
I20240426 13:20:52.549383 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:52.553063 25023 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:52.506628 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.499042 28870 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.465545 29402 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.521013 29566 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.486538 29558 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.022s
I20240426 13:20:52.525714 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:20:52.528287 28899 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.835s	user 0.010s	sys 0.007s
W20240426 13:20:52.528524 29253 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.105s	user 0.007s	sys 0.010s
I20240426 13:20:52.511967 29503 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.498054 29568 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.529917 29359 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.530189 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 586) took 1664 ms. Trace:
W20240426 13:20:52.556432 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49878.3us
I20240426 13:20:52.565241 24950 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:52.571993 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.567535 29534 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:52.571285 29546 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:20:52.574926 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49295.9us
I20240426 13:20:52.609090 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.588596 25621 rpcz_store.cc:276] 0426 13:20:50.865425 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.865547 (+   122us) service_pool.cc:225] Handling call
0426 13:20:52.530012 (+1664465us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.530139 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1084683648}
I20240426 13:20:52.579439 29560 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.617476 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:52.617404 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.620427 24955 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:52.580899 29571 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:52.620561 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:52.622812 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 73510.7us
I20240426 13:20:52.612931 29509 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.626286 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.623924 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.624650 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 787) took 1213 ms. Trace:
W20240426 13:20:52.622104 29442 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.961s	user 0.014s	sys 0.007s
I20240426 13:20:52.628242 25615 rpcz_store.cc:276] 0426 13:20:51.411019 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.411185 (+   166us) service_pool.cc:225] Handling call
0426 13:20:51.411554 (+   369us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:51.440530 (+ 28976us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:20:51.805164 (+364634us) log.cc:842] Serialized 14 byte log entry
0426 13:20:52.393823 (+588659us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:52.436081 (+ 42258us) log.cc:842] Serialized 26 byte log entry
0426 13:20:52.436254 (+   173us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:52.436476 (+   222us) log.cc:842] Serialized 14 byte log entry
0426 13:20:52.436629 (+   153us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.436699 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:52.624470 (+187771us) raft_consensus.cc:1671] finished
0426 13:20:52.624489 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:52.624598 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58218,"prepare.queue_time_us":292542,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":67,"raft.queue_time_us":136224,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":13348,"spinlock_wait_cycles":7186688,"thread_start_us":765138,"threads_started":3,"wal-append.queue_time_us":302349}
I20240426 13:20:52.587940 28899 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.588303 29253 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.511952 29556 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.623602 29359 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:52.714483 29494 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:52.714605 29359 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:52.715004 29494 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:52.715101 29359 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:52.616286 29413 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20240426 13:20:52.717737 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 768) took 1849 ms. Trace:
I20240426 13:20:52.718101 24952 rpcz_store.cc:276] 0426 13:20:50.868378 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.147637 (+1279259us) service_pool.cc:225] Handling call
0426 13:20:52.147990 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.176892 (+ 28902us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.717537 (+540645us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x7b540038efd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:52.717685 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17512320}
I20240426 13:20:52.719223 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.719877 24952 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:52.706581 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.707087 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.706581 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:52.710914 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:52.632571 29401 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20240426 13:20:52.627831 28758 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:52.627560 29582 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:52.711580 29576 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240426 13:20:52.764804 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 213581us
W20240426 13:20:52.713320 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2445) took 1302 ms (client timeout 1698 ms). Trace:
W20240426 13:20:52.765813 24949 rpcz_store.cc:269] 0426 13:20:51.410633 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.494310 (+ 83677us) service_pool.cc:225] Handling call
0426 13:20:52.713169 (+1218859us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.713262 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220815616}
I20240426 13:20:52.719972 29253 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.720417 25026 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:52.720441 24948 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:52.631982 25100 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20240426 13:20:52.721163 28899 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.630260 29574 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:52.721436 29378 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:52.722467 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2439) took 1316 ms (client timeout 1561 ms). Trace:
W20240426 13:20:52.771580 24953 rpcz_store.cc:269] 0426 13:20:51.406143 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.492244 (+ 86101us) service_pool.cc:225] Handling call
0426 13:20:52.722298 (+1230054us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.722415 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1558600960}
I20240426 13:20:52.771929 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:52.772969 28906 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.616s	user 0.002s	sys 0.007s
I20240426 13:20:52.773325 28906 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.723062 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1817) took 1459 ms. Trace:
I20240426 13:20:52.723845 29494 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.726346 29359 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.727631 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.713320 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2586) took 1039 ms. Trace:
W20240426 13:20:52.732550 29577 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20240426 13:20:52.733304 29408 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.733930 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.736078 29351 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:52.737584 29398 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:52.738863 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1823) took 1438 ms. Trace:
I20240426 13:20:52.783658 25031 rpcz_store.cc:276] 0426 13:20:51.300374 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.351636 (+ 51262us) service_pool.cc:225] Handling call
0426 13:20:52.738659 (+1387023us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54000b0754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.738801 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1691376896}
I20240426 13:20:52.736279 29058 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
W20240426 13:20:52.744745 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 738) took 2410 ms (client timeout 1697 ms). Trace:
I20240426 13:20:52.747434 29365 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.623355 29571 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:52.749173 29568 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:52.751794 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45285.6us
I20240426 13:20:52.760651 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.762907 25022 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:52.632297 29534 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
W20240426 13:20:52.708571 29570 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:20:52.701182 29442 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.627406 28870 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:52.769560 29388 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.033s	user 0.010s	sys 0.000s
I20240426 13:20:52.775928 24952 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.6-13.6]
I20240426 13:20:52.777140 25101 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:52.774502 25024 rpcz_store.cc:276] 0426 13:20:51.263969 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.264102 (+   133us) service_pool.cc:225] Handling call
0426 13:20:52.722905 (+1458803us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.723015 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242756864}
W20240426 13:20:52.778134 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 150367us
I20240426 13:20:52.804821 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.805461 25022 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:52.778235 25025 rpcz_store.cc:276] 0426 13:20:51.674302 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.678862 (+  4560us) service_pool.cc:225] Handling call
0426 13:20:52.713169 (+1034307us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.713270 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1669760768}
I20240426 13:20:52.761395 29562 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:52.773991 29008 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.761152 29413 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20240426 13:20:52.835995 29447 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.837021 29008 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:52.837749 29017 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.838165 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.841568 29418 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.797659 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:52.785624 25099 rpcz_store.cc:269] 0426 13:20:50.334727 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.927725 (+592998us) service_pool.cc:225] Handling call
0426 13:20:52.744580 (+1816855us) spinlock_profiling.cc:243] Waited 8.98 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.744681 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20644224}
I20240426 13:20:52.799326 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.779917 29574 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:52.847630 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50066.7us
I20240426 13:20:52.848331 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.849681 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.803570 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.782382 29582 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.805617 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2003) took 1715 ms. Trace:
I20240426 13:20:52.851817 24952 rpcz_store.cc:276] 0426 13:20:51.090336 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.752521 (+1662185us) service_pool.cc:225] Handling call
0426 13:20:52.775744 (+ 23223us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:52.805377 (+ 29633us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.805570 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:52.801977 29503 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.801856 29534 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:52.794457 29572 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:52.817299 28906 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:52.818434 29359 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.819046 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.819479 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.749581 29566 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.820225 29494 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.821672 29442 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.821995 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2573) took 1213 ms. Trace:
I20240426 13:20:52.808653 29576 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:52.794180 29351 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.863914 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:52.861891 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:52.861255 25029 rpcz_store.cc:276] 0426 13:20:51.608853 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.944155 (+335302us) service_pool.cc:225] Handling call
0426 13:20:52.821855 (+877700us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.821946 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":743886720}
I20240426 13:20:52.862440 25022 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:52.857587 29562 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:52.824785 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.825026 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.797375 29571 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:52.825681 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:52.826081 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:52.826558 29500 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.603s	user 0.008s	sys 0.001s
I20240426 13:20:52.827265 29370 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:52.826568 29058 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:52.829514 25617 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
W20240426 13:20:52.830145 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 819) took 2177 ms (client timeout 1512 ms). Trace:
I20240426 13:20:52.863847 29576 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.783562 29560 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.799435 29366 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.833174 25101 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:52.833505 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 769) took 1964 ms. Trace:
I20240426 13:20:52.904177 24950 rpcz_store.cc:276] 0426 13:20:50.868658 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.452947 (+1584289us) service_pool.cc:225] Handling call
0426 13:20:52.468882 (+ 15935us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.605884 (+137002us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.833303 (+227419us) spinlock_profiling.cc:243] Waited 96.1 ms on lock 0x7b54001d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:52.833450 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221062400}
I20240426 13:20:52.834261 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 774) took 1603 ms. Trace:
I20240426 13:20:52.905786 24947 rpcz_store.cc:276] 0426 13:20:51.231059 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.309257 (+ 78198us) service_pool.cc:225] Handling call
0426 13:20:52.834118 (+1524861us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.834211 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352755968}
I20240426 13:20:52.834458 25100 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:52.808925 29388 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.909538 29366 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.910441 29388 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.800114 29563 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
W20240426 13:20:52.911464 29539 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.820s	user 0.003s	sys 0.000s
I20240426 13:20:52.855427 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.912227 25031 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:52.858014 24955 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:52.913232 29583 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20240426 13:20:52.915478 24947 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:52.915879 29252 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:20:52.916186 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 771) took 1686 ms. Trace:
I20240426 13:20:52.916728 24947 rpcz_store.cc:276] 0426 13:20:51.230143 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.906661 (+1676518us) service_pool.cc:225] Handling call
0426 13:20:52.915272 (+  8611us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.915930 (+   658us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.916132 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:52.917927 29534 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.918462 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2010) took 1686 ms. Trace:
I20240426 13:20:52.918972 24947 rpcz_store.cc:276] 0426 13:20:51.231916 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.917568 (+1685652us) service_pool.cc:225] Handling call
0426 13:20:52.917903 (+   335us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.918090 (+   187us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:52.918124 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:52.918173 (+    49us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:52.918221 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.918279 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:52.918405 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:52.920208 29582 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:52.922451 29515 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.246s	user 0.002s	sys 0.000s
W20240426 13:20:52.923821 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 821) took 2243 ms (client timeout 1872 ms). Trace:
I20240426 13:20:52.862890 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.859896 29568 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:52.870510 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.871337 25025 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:52.823498 29379 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:52.872057 25619 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:52.861482 29503 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:52.953239 24952 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:52.933400 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:52.872361 28758 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:52.880265 29500 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.883567 29571 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.959460 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:52.884209 29370 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:52.885282 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60565.6us
I20240426 13:20:52.961683 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:52.886083 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60473.4us
I20240426 13:20:52.886751 29494 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:52.887202 29587 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:52.889883 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2004) took 1705 ms. Trace:
I20240426 13:20:52.896899 29574 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:52.900442 25616 rpcz_store.cc:269] 0426 13:20:50.652503 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.652647 (+   144us) service_pool.cc:225] Handling call
0426 13:20:52.829932 (+2177285us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54006b26d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.830058 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":641835776}
I20240426 13:20:52.831718 29094 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:52.910756 29090 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.911607 29563 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
W20240426 13:20:52.929677 25618 rpcz_store.cc:269] 0426 13:20:50.680771 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.680917 (+   146us) service_pool.cc:225] Handling call
0426 13:20:52.923600 (+2242683us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400473954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.923740 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821342976}
I20240426 13:20:52.977305 29563 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.978202 29563 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:52.979450 29563 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:52.984310 29366 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:52.984714 29366 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:52.985006 29366 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:52.928040 29481 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20240426 13:20:52.990679 29571 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:52.930399 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 67564.5us
I20240426 13:20:52.991479 29500 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:52.925463 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 581) took 2061 ms. Trace:
I20240426 13:20:52.991850 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.992306 25622 rpcz_store.cc:276] 0426 13:20:50.864022 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.864165 (+   143us) service_pool.cc:225] Handling call
0426 13:20:50.876200 (+ 12035us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:51.540583 (+664383us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.925155 (+1384572us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:52.925394 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1525873408}
W20240426 13:20:52.926545 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 766) took 3946 ms (client timeout 3126 ms). Trace:
W20240426 13:20:52.993580 24951 rpcz_store.cc:269] 0426 13:20:48.980432 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:48.980570 (+   138us) service_pool.cc:225] Handling call
0426 13:20:52.926324 (+3945754us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400078fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:52.926448 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3824110464}
I20240426 13:20:52.994765 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:52.931757 24950 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:52.927856 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:52.957726 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:52.949810 29379 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:52.996205 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:52.996467 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.959314 25100 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:52.965277 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.996850 29379 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:52.997666 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.997870 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.916253 29252 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:52.998752 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:52.999532 29252 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.000488 29370 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:53.000735 29252 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:52.935776 29388 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:52.902791 29058 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.004549 29388 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.005143 29388 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:53.005944 29388 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:53.006143 29058 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:53.006299 29388 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:52.955612 29293 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
W20240426 13:20:53.007099 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 959750us
I20240426 13:20:52.924345 29351 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:52.932941 29557 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:53.008625 29557 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.009920 29446 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:53.010448 29557 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:53.011154 29430 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.011392 25098 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:53.011979 29430 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.012595 25099 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:53.013723 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.014292 25096 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:53.015385 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2006) took 1817 ms. Trace:
I20240426 13:20:53.015771 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:53.015760 24952 rpcz_store.cc:276] 0426 13:20:51.197655 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.868522 (+1670867us) service_pool.cc:225] Handling call
0426 13:20:52.880555 (+ 12033us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.953766 (+ 73211us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.015174 (+ 61408us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:53.015330 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":166092544}
I20240426 13:20:52.998551 29379 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:53.000512 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:53.015547 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 918487us
I20240426 13:20:53.026183 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:52.998847 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:52.970752 24949 rpcz_store.cc:276] 0426 13:20:51.184600 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.766912 (+1582312us) service_pool.cc:225] Handling call
0426 13:20:52.768918 (+  2006us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:52.769394 (+   476us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:52.769437 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:52.770131 (+   694us) log.cc:842] Serialized 52 byte log entry
0426 13:20:52.794094 (+ 23963us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:52.869489 (+ 75395us) log.cc:842] Serialized 14 byte log entry
0426 13:20:52.869893 (+   404us) log.cc:842] Serialized 14 byte log entry
0426 13:20:52.870051 (+   158us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:52.870112 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:52.889625 (+ 19513us) raft_consensus.cc:1671] finished
0426 13:20:52.889653 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:52.889813 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3046,"raft.queue_time_us":6872,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":28989,"thread_start_us":90229,"threads_started":2,"wal-append.queue_time_us":90735}
I20240426 13:20:52.995018 24951 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 4
I20240426 13:20:53.027493 24951 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20240426 13:20:52.967006 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:53.028726 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61776.9us
I20240426 13:20:53.029493 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.027841 24951 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:53.023922 25025 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:20:53.034122 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12.
I20240426 13:20:53.034601 24921 leader_election.cc:401] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:53.038030 28870 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.039113 25022 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:53.039139 24952 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20240426 13:20:53.040647 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2016) took 1789 ms. Trace:
I20240426 13:20:53.040979 24952 rpcz_store.cc:276] 0426 13:20:51.255982 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.016872 (+1760890us) service_pool.cc:225] Handling call
0426 13:20:53.038920 (+ 22048us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:53.040393 (+  1473us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.040595 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:53.024546 29490 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:52.926661 29576 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.043565 29576 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:53.045946 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.046367 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.046888 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.047763 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.048010 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.049531 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.050176 25105 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:53.007326 29388 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.057334 25024 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:20:53.046352 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.060254 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:53.060807 25615 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:53.062340 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.067127 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:53.064323 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.069012 24949 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:53.069725 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 776) took 1813 ms. Trace:
I20240426 13:20:53.070302 25105 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:53.071187 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:53.059664 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.074071 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:53.074287 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.078094 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:20:53.078186 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:53.081025 25622 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:53.024518 29090 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:53.083878 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.070077 24949 rpcz_store.cc:276] 0426 13:20:51.256751 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.027303 (+1770552us) service_pool.cc:225] Handling call
0426 13:20:53.068708 (+ 41405us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.069463 (+   755us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.069663 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:52.967846 29413 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.105s
I20240426 13:20:53.063357 28870 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:53.065929 29500 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 12
I20240426 13:20:53.100471 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2001) took 2091 ms. Trace:
I20240426 13:20:53.100235 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.061683 29587 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:53.067224 29429 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
W20240426 13:20:53.105556 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63082.2us
I20240426 13:20:53.106501 25624 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:53.107443 25101 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:53.109608 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 181824us
I20240426 13:20:53.070794 29351 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.121078 25103 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:53.124296 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.124476 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.116070 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.126734 25025 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20240426 13:20:53.130275 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:53.100306 29370 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:53.136482 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.084304 29090 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:53.130390 24949 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:53.131433 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2580) took 1515 ms. Trace:
W20240426 13:20:53.143400 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117272us
I20240426 13:20:53.143744 25028 rpcz_store.cc:276] 0426 13:20:51.616358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.207831 (+591473us) service_pool.cc:225] Handling call
0426 13:20:52.208052 (+   221us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.120496 (+912444us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.120527 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:53.131128 (+ 10601us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:53.131192 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.131270 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.131381 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":912088,"raft.queue_time_us":912360,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9700}
I20240426 13:20:53.100364 29005 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:53.137312 25025 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 14
I20240426 13:20:53.103703 29571 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:53.126369 29421 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:53.141199 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.108256 29388 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:53.141990 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.150947 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.152292 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.152925 25022 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:53.105952 28870 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:53.156292 28870 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:53.162169 29563 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:53.164613 29563 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:20:53.166136 29567 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:53.076313 29058 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
W20240426 13:20:53.136540 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1826) took 1835 ms (client timeout 1555 ms). Trace:
W20240426 13:20:53.174151 25027 rpcz_store.cc:269] 0426 13:20:51.301454 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.310285 (+  8831us) service_pool.cc:225] Handling call
0426 13:20:53.136401 (+1826116us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:53.136486 (+    85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":658649472}
I20240426 13:20:53.103869 29568 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.145386 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.170125 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:53.170862 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 160644us
I20240426 13:20:53.171311 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.144343 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.177698 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.177604 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.177520 25028 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:53.142557 29351 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:53.180742 25031 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:53.182796 25027 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:20:53.181270 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.144649 29090 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:53.139783 29429 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:53.194265 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:53.185916 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.169353 29571 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:53.199759 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1994) took 2282 ms. Trace:
I20240426 13:20:53.195919 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.201758 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:53.202347 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.140496 29016 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20240426 13:20:53.201534 24954 rpcz_store.cc:276] 0426 13:20:50.917556 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.361044 (+443488us) service_pool.cc:225] Handling call
0426 13:20:53.199514 (+1838470us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54000a73d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:53.199691 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1833917312}
I20240426 13:20:53.211863 25615 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:53.215888 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:53.206037 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:53.182760 29587 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:53.209070 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.249357 24954 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:53.247341 25022 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:53.182592 29510 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:53.176862 29005 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:53.193948 29451 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:53.252382 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.149854 29481 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.055s
I20240426 13:20:53.253054 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.253865 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:53.198493 29220 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:53.205588 29571 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:53.238184 29278 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.177071 29421 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.203270 29425 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.183598 29582 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.073s
I20240426 13:20:53.282655 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.252189 29510 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:53.204097 29351 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:53.251478 29534 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:53.238456 29556 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20240426 13:20:53.313612 25024 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:53.281385 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:53.285892 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79925.6us
I20240426 13:20:53.315050 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.315603 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.260679 29451 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:53.197947 29479 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20240426 13:20:53.285856 29220 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.258471 29511 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:53.314164 25024 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.317804 29253 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:53.204021 29413 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20240426 13:20:53.284596 29051 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:53.351295 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:53.352336 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:53.353081 25618 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:20:53.325595 25615 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20240426 13:20:53.316920 29278 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:20:53.339398 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:53.343350 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 173241us
I20240426 13:20:53.295634 29510 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:53.321346 29451 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:53.318897 28906 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.356267 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:53.296916 29538 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20240426 13:20:53.356943 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:53.373499 25622 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:53.338251 29534 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:53.392339 29534 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.322228 29094 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.398695 25096 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:53.355695 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:53.399781 25617 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:53.321328 29425 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:53.372712 29595 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
W20240426 13:20:53.375840 29599 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20240426 13:20:53.376243 29220 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:53.376873 29596 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20240426 13:20:53.378509 29253 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:53.429028 29253 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.431416 29253 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:53.381309 29500 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12.
I20240426 13:20:53.283617 29005 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.388515 29421 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:53.389226 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.436424 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.437294 25022 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:53.389128 29351 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.396648 25101 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:53.397466 25030 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:53.362586 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:53.441385 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78953.8us
I20240426 13:20:53.441958 29500 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.400331 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2015) took 2153 ms. Trace:
I20240426 13:20:53.338855 29447 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20240426 13:20:53.395172 29584 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:53.444242 24951 rpcz_store.cc:276] 0426 13:20:51.246687 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.994433 (+1747746us) service_pool.cc:225] Handling call
0426 13:20:52.994810 (+   377us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:53.030881 (+ 36071us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.030925 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:53.031501 (+   576us) log.cc:842] Serialized 52 byte log entry
0426 13:20:53.321747 (+290246us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:53.321788 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.321853 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:53.399534 (+ 77681us) raft_consensus.cc:1671] finished
0426 13:20:53.399575 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.400243 (+   668us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":209,"raft.queue_time_us":904,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":14045,"thread_start_us":216528,"threads_started":1,"wal-append.queue_time_us":309952}
I20240426 13:20:53.428349 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2005) took 2238 ms. Trace:
I20240426 13:20:53.446542 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:53.447520 24951 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:53.445590 29584 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:53.446827 24953 rpcz_store.cc:276] 0426 13:20:51.190168 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.772399 (+1582231us) service_pool.cc:225] Handling call
0426 13:20:52.772802 (+   403us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:52.773266 (+   464us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:52.773305 (+    39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:52.773937 (+   632us) log.cc:842] Serialized 52 byte log entry
0426 13:20:53.066576 (+292639us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:53.100609 (+ 34033us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.100973 (+   364us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.104541 (+  3568us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.104677 (+   136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:53.428094 (+323417us) raft_consensus.cc:1671] finished
0426 13:20:53.428118 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.428286 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3223,"prepare.queue_time_us":74310,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":5734,"raft.queue_time_us":7788,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":4721,"spinlock_wait_cycles":9796224,"thread_start_us":292186,"threads_started":1,"wal-append.queue_time_us":425256}
I20240426 13:20:53.450284 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.448769 29584 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.453346 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:53.348922 29509 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20240426 13:20:53.321688 29429 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.362371 29511 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.460331 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.456183 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.360011 29366 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.472488 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:53.455578 29584 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:53.458242 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2011) took 2226 ms. Trace:
I20240426 13:20:53.422555 29568 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:53.463836 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 282341us
I20240426 13:20:53.464497 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:53.452077 29567 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:53.475327 25025 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: [10.9-11.10]
I20240426 13:20:53.457903 29359 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:53.451433 29430 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240426 13:20:53.460618 29554 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:53.457783 29425 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.500319 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:53.476079 29421 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20240426 13:20:53.509145 29430 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:53.509243 29421 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:53.509739 29430 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.509912 29421 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.510866 29430 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.511097 29421 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.523834 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 775) took 2292 ms. Trace:
I20240426 13:20:53.535406 24955 rpcz_store.cc:276] 0426 13:20:51.231358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.913268 (+1681910us) service_pool.cc:225] Handling call
0426 13:20:52.913587 (+   319us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:52.913807 (+   220us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:52.913835 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:52.914277 (+   442us) log.cc:842] Serialized 26 byte log entry
0426 13:20:53.061011 (+146734us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:53.061841 (+   830us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.061905 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:53.523550 (+461645us) raft_consensus.cc:1671] finished
0426 13:20:53.523581 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.523757 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":146371,"threads_started":1,"wal-append.queue_time_us":269410}
I20240426 13:20:53.537039 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.537119 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.487246 29568 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.541211 25101 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:53.541703 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.489315 29500 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.542112 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.548610 24953 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:53.495450 29359 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:53.559335 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.558068 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.596434 24951 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:53.593849 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.560875 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:53.601707 24953 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:53.609786 29423 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:53.610736 25029 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:20:53.612455 29374 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:53.611930 29359 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:53.616623 29610 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20240426 13:20:53.616689 29009 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:53.617302 29009 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:53.618197 29009 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.618559 29503 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:53.495362 29367 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.065s
W20240426 13:20:53.620998 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47181.7us
I20240426 13:20:53.618607 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:53.554857 29481 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:20:53.640832 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 780) took 1791 ms. Trace:
I20240426 13:20:53.553709 29430 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:53.553156 29220 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:53.658931 24953 rpcz_store.cc:276] 0426 13:20:51.849237 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.602515 (+1753278us) service_pool.cc:225] Handling call
0426 13:20:53.618316 (+ 15801us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.640456 (+ 22140us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.640759 (+   303us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:53.621186 29252 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.620s	user 0.005s	sys 0.004s
I20240426 13:20:53.654709 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1291) took 1696 ms. Trace:
I20240426 13:20:53.671813 25102 rpcz_store.cc:276] 0426 13:20:51.924314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.017053 (+ 92739us) service_pool.cc:225] Handling call
0426 13:20:53.619848 (+1602795us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:53.620379 (+   531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":701996800}
I20240426 13:20:53.602084 29421 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:53.673938 25098 raft_consensus.cc:2396] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20240426 13:20:53.677215 25104 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
W20240426 13:20:53.681876 24995 connection.cc:232] RPC call timeout handler was delayed by -1.0246250559999908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20240426 13:20:53.690246 25102 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:53.688894 24953 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20240426 13:20:53.690914 24953 pending_rounds.cc:76] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:20:53.691134 24953 pending_rounds.cc:97] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20240426 13:20:53.691418 24953 raft_consensus.cc:2866] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:53.692127 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 782) took 1235 ms. Trace:
I20240426 13:20:53.692397 24953 rpcz_store.cc:276] 0426 13:20:52.456745 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.666374 (+1209629us) service_pool.cc:225] Handling call
0426 13:20:53.688560 (+ 22186us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.691881 (+  3321us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.692079 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:53.693300 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:53.688620 29425 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.619325 29503 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:53.701354 25028 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:20:53.665819 29509 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:53.620018 29447 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
I20240426 13:20:53.711938 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:20:53.713335 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.641250 29451 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.717779 29462 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.013s
I20240426 13:20:53.732656 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2007) took 2523 ms. Trace:
I20240426 13:20:53.733582 29451 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:53.733167 24950 rpcz_store.cc:276] 0426 13:20:51.209443 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.905243 (+1695800us) service_pool.cc:225] Handling call
0426 13:20:52.908591 (+  3348us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.995510 (+ 86919us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.732339 (+736829us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b540030c5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:53.732586 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52678528}
I20240426 13:20:53.746356 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:20:53.746703 24995 leader_election.cc:401] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:53.687609 29220 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:53.748013 24955 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:53.720470 29366 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:53.719753 29279 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:53.771112 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2448) took 1270 ms. Trace:
I20240426 13:20:53.775100 24950 rpcz_store.cc:276] 0426 13:20:52.500557 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.744593 (+1244036us) service_pool.cc:225] Handling call
0426 13:20:53.767615 (+ 23022us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.767813 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.767849 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:53.767901 (+    52us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:53.767945 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.768009 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.771047 (+  3038us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:53.106436 24956 rpcz_store.cc:276] 0426 13:20:51.009205 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.547782 (+1538577us) service_pool.cc:225] Handling call
0426 13:20:52.571294 (+ 23512us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:52.579259 (+  7965us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:52.579305 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:52.579800 (+   495us) log.cc:842] Serialized 26 byte log entry
0426 13:20:52.780332 (+200532us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:53.081465 (+301133us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.100001 (+ 18536us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.100140 (+   139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:53.100215 (+    75us) raft_consensus.cc:1671] finished
0426 13:20:53.100232 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.100397 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18265,"prepare.queue_time_us":173124,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":2862,"raft.queue_time_us":18578,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":10850,"thread_start_us":442393,"threads_started":2,"wal-append.queue_time_us":258061}
W20240426 13:20:53.761873 28870 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.605s	user 0.008s	sys 0.003s
I20240426 13:20:53.760078 29447 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.708714 29218 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
W20240426 13:20:53.782938 29379 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.739s	user 0.010s	sys 0.000s
W20240426 13:20:53.797894 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 342505us
I20240426 13:20:53.798305 24950 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:20:53.798636 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.769193 29587 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:20:53.793252 29616 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.240s	user 0.000s	sys 0.000s
I20240426 13:20:53.766971 29509 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:20:53.807088 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:53.801059 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2450) took 1289 ms. Trace:
I20240426 13:20:53.808532 24950 rpcz_store.cc:276] 0426 13:20:52.511780 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.776966 (+1265186us) service_pool.cc:225] Handling call
0426 13:20:53.795425 (+ 18459us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.798652 (+  3227us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.800993 (+  2341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:53.804534 25617 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:53.820214 25622 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:53.783474 29511 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:53.803759 29612 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20240426 13:20:53.826860 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:53.827644 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 219311us
I20240426 13:20:53.774214 29451 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.045s
I20240426 13:20:53.848280 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.857336 24950 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:53.860921 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2455) took 1284 ms. Trace:
I20240426 13:20:53.861284 24950 rpcz_store.cc:276] 0426 13:20:52.576159 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.809437 (+1233278us) service_pool.cc:225] Handling call
0426 13:20:53.857142 (+ 47705us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.860658 (+  3516us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.860862 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:53.822836 29576 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.779s	user 0.006s	sys 0.006s
I20240426 13:20:53.824000 29016 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.904165 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:53.859467 29379 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.904856 25031 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:53.904943 29537 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:53.905700 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:53.905925 29537 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:20:53.906567 29537 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240426 13:20:53.909826 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.911078 29628 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:53.911571 29628 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:53.912012 29628 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:53.822691 29005 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:53.817510 29462 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
W20240426 13:20:53.894361 29510 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.503s	user 0.009s	sys 0.009s
W20240426 13:20:53.896456 29571 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.601s	user 0.007s	sys 0.002s
I20240426 13:20:53.858007 29220 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.823544 29568 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:53.906183 25031 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:53.914633 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:53.914830 25096 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20240426 13:20:53.988330 25096 pending_rounds.cc:76] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 10
I20240426 13:20:53.988694 25096 pending_rounds.cc:97] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20240426 13:20:53.988988 25096 raft_consensus.cc:2866] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:53.893625 29503 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:53.892283 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:53.916863 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:53.918850 25616 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:20:53.799559 28870 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:53.842304 29351 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:53.877727 29451 consensus_queue.cc:1225] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:20:53.903789 29374 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:20:53.862273 29218 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:53.948881 24950 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:53.817936 29278 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20240426 13:20:53.862452 29554 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:53.990248 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:53.991034 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:54.038321 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47397.9us
I20240426 13:20:54.039614 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:54.043952 25105 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20240426 13:20:54.045492 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 331721us
I20240426 13:20:54.047510 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:54.082866 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:54.135444 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2575) took 2520 ms. Trace:
I20240426 13:20:54.135808 25023 rpcz_store.cc:276] 0426 13:20:51.614971 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.199397 (+584426us) service_pool.cc:225] Handling call
0426 13:20:52.199741 (+   344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.564591 (+364850us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.135184 (+1570593us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:54.135387 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":793819520}
I20240426 13:20:53.944242 29510 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:54.272209 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133528us
W20240426 13:20:54.275789 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.023s	sys 0.018s
W20240426 13:20:54.276243 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.024s	sys 0.019s
W20240426 13:20:54.276548 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 528953us
W20240426 13:20:54.331182 25025 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.018s	sys 0.001s
W20240426 13:20:54.331828 25025 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.018s	sys 0.001s
I20240426 13:20:54.349385 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:53.979375 29571 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:54.468835 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 119572us
W20240426 13:20:54.373235 29430 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.705s	user 0.018s	sys 0.003s
I20240426 13:20:54.373991 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:54.485252 25023 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:54.025276 29379 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.261s
I20240426 13:20:54.135190 29534 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:54.389997 25028 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:54.388239 29005 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.510681 29005 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:54.415563 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 551201us
I20240426 13:20:54.023201 29351 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.036219 29218 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.350262 29094 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.083247 29374 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.307s
I20240426 13:20:54.374150 29366 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.052s
I20240426 13:20:54.424232 29630 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:54.394042 29614 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:54.424070 29617 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20240426 13:20:54.419190 29503 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:54.430589 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 601) took 1179 ms. Trace:
W20240426 13:20:54.434374 29421 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.750s	user 0.009s	sys 0.000s
W20240426 13:20:54.435920 29388 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.263s	user 0.010s	sys 0.000s
I20240426 13:20:54.436252 29447 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.436561 29634 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:54.529381 29094 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.554158 29634 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:54.555291 29634 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.556591 29634 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:54.556309 29503 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.557997 29366 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:54.558501 29366 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:54.437136 29615 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.656s	user 0.000s	sys 0.000s
I20240426 13:20:54.350750 29554 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:54.559450 29366 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:54.456845 29090 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.252s	user 0.008s	sys 0.002s
I20240426 13:20:54.560823 29090 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.481990 25031 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:54.452731 29644 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:20:54.562964 29644 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:54.563956 29644 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.498296 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:54.565237 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2437) took 3160 ms. Trace:
W20240426 13:20:54.499766 29640 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20240426 13:20:54.566159 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:54.566955 25023 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:54.500635 29633 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.568580 29633 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.571090 29633 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.565725 24952 rpcz_store.cc:276] 0426 13:20:51.404895 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.042327 (+1637432us) service_pool.cc:225] Handling call
0426 13:20:53.042702 (+   375us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:53.043145 (+   443us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.043173 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:53.043686 (+   513us) log.cc:842] Serialized 52 byte log entry
0426 13:20:53.495023 (+451337us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:53.552559 (+ 57536us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.552916 (+   357us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.552999 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:54.564977 (+1011978us) raft_consensus.cc:1671] finished
0426 13:20:54.565003 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.565174 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":584,"prepare.queue_time_us":57667,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":9128,"raft.queue_time_us":83190,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":6235,"spinlock_wait_cycles":47570688,"thread_start_us":442723,"threads_started":1,"wal-append.queue_time_us":703457}
I20240426 13:20:54.522362 24954 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20240426 13:20:54.554003 29447 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.447644 29510 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.449291 28870 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:54.569029 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:54.546025 25022 rpcz_store.cc:276] 0426 13:20:53.251089 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.437848 (+186759us) service_pool.cc:225] Handling call
0426 13:20:53.438204 (+   356us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:53.474358 (+ 36154us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.474402 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:53.483752 (+  9350us) log.cc:842] Serialized 52 byte log entry
0426 13:20:53.823259 (+339507us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:53.915443 (+ 92184us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.922762 (+  7319us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.922857 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:54.430367 (+507510us) raft_consensus.cc:1671] finished
0426 13:20:54.430388 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.430532 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53451,"prepare.queue_time_us":40617,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":45841,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":27619,"thread_start_us":314458,"threads_started":1,"wal-append.queue_time_us":644496}
I20240426 13:20:54.574154 24952 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:54.581352 25023 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:54.509508 29628 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.597s	user 0.008s	sys 0.000s
I20240426 13:20:54.650250 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2447) took 2151 ms. Trace:
I20240426 13:20:54.521509 29630 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:54.555763 29094 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.494714 29509 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:20:54.656251 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 784) took 1934 ms. Trace:
I20240426 13:20:54.763337 24952 rpcz_store.cc:276] 0426 13:20:52.722330 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.573002 (+1850672us) service_pool.cc:225] Handling call
0426 13:20:54.573347 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.653323 (+ 79976us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.656186 (+  2863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.715927 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2613) took 1039 ms. Trace:
I20240426 13:20:54.764531 25023 rpcz_store.cc:276] 0426 13:20:53.676264 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.580822 (+904558us) service_pool.cc:225] Handling call
0426 13:20:54.581159 (+   337us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.715603 (+134444us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.715863 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.662751 24953 rpcz_store.cc:276] 0426 13:20:52.498728 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.693445 (+1194717us) service_pool.cc:225] Handling call
0426 13:20:53.694082 (+   637us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:53.703253 (+  9171us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.703316 (+    63us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:53.717404 (+ 14088us) log.cc:842] Serialized 78 byte log entry
0426 13:20:53.980522 (+263118us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:54.022244 (+ 41722us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.035873 (+ 13629us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.036160 (+   287us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.036299 (+   139us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.036381 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:54.650006 (+613625us) raft_consensus.cc:1671] finished
0426 13:20:54.650026 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.650191 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":401097,"prepare.queue_time_us":44951,"prepare.run_wall_time_us":39,"raft.queue_time_us":33751,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":130962,"spinlock_wait_cycles":438097408,"thread_start_us":204254,"threads_started":1,"wal-append.queue_time_us":680998}
I20240426 13:20:54.715157 29628 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.769099 29017 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:54.574554 29218 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.548869 29534 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:54.650056 29366 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:54.585681 29571 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.762801 29094 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.749342 29630 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:54.548245 29614 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.588297 29554 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.768720 29351 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.767740 29005 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.776643 25031 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:20:54.616390 29374 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:54.760937 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.023s	sys 0.053s
I20240426 13:20:54.770923 25023 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20240426 13:20:54.654496 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 184645us
I20240426 13:20:54.571763 29633 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.194s
W20240426 13:20:54.784176 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.025s	sys 0.021s
I20240426 13:20:54.789994 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 603) took 1530 ms. Trace:
I20240426 13:20:54.794994 24952 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20240426 13:20:54.780654 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.028s	sys 0.049s
I20240426 13:20:54.789404 29629 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.789625 29017 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:54.947291 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2617) took 1202 ms. Trace:
W20240426 13:20:54.949288 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.030s	sys 0.030s
W20240426 13:20:54.950081 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.030s	sys 0.055s
W20240426 13:20:54.860664 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.014s	sys 0.012s
I20240426 13:20:54.882473 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2454) took 2314 ms. Trace:
I20240426 13:20:54.949537 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2614) took 1258 ms. Trace:
W20240426 13:20:54.949689 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.006s	sys 0.033s
I20240426 13:20:54.948686 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 785) took 2226 ms. Trace:
I20240426 13:20:54.900157 28777 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:54.949270 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.034s	sys 0.061s
W20240426 13:20:54.956295 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.020s	sys 0.016s
W20240426 13:20:54.956143 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 916554us
I20240426 13:20:54.956360 24956 rpcz_store.cc:276] 0426 13:20:52.568390 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.778404 (+1210014us) service_pool.cc:225] Handling call
0426 13:20:53.778815 (+   411us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:53.779235 (+   420us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.779295 (+    60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:53.789690 (+ 10395us) log.cc:842] Serialized 26 byte log entry
0426 13:20:54.281376 (+491686us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:54.881399 (+600023us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.882003 (+   604us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.882105 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:54.882165 (+    60us) raft_consensus.cc:1671] finished
0426 13:20:54.882234 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.882406 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":488669,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":192,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":23588,"thread_start_us":906708,"threads_started":2,"wal-append.queue_time_us":639021}
W20240426 13:20:54.958880 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.09677e+06us
I20240426 13:20:54.948684 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:54.956707 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.007s	sys 0.033s
I20240426 13:20:54.956048 25023 rpcz_store.cc:276] 0426 13:20:53.744988 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.770238 (+1025250us) service_pool.cc:225] Handling call
0426 13:20:54.770569 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.947027 (+176458us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.947230 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.961712 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:54.832361 29388 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.956594 25031 rpcz_store.cc:276] 0426 13:20:53.690678 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.714004 (+1023326us) service_pool.cc:225] Handling call
0426 13:20:54.776337 (+ 62333us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.949263 (+172926us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.949451 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:54.871968 29557 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.043s	user 0.017s	sys 0.041s
I20240426 13:20:54.964318 25619 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:54.962055 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:54.965366 25105 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:54.957355 24952 rpcz_store.cc:276] 0426 13:20:52.722611 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.778661 (+2056050us) service_pool.cc:225] Handling call
0426 13:20:54.794600 (+ 15939us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.948416 (+153816us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.948631 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.799129 29534 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.949395 25025 rpcz_store.cc:276] 0426 13:20:53.259826 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.490191 (+230365us) service_pool.cc:225] Handling call
0426 13:20:53.490476 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.490759 (+   283us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:54.332384 (+841625us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.780152 (+447768us) log.cc:842] Serialized 14 byte log entry
0426 13:20:54.789685 (+  9533us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:54.789745 (+    60us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:54.789792 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.789851 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.789950 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9247,"prepare.queue_time_us":55543,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":9434,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":9089,"spinlock_wait_cycles":4085504,"thread_start_us":1228401,"threads_started":2,"wal-append.queue_time_us":559800}
I20240426 13:20:54.982532 25100 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:54.961333 29220 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:54.945905 29587 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:54.988333 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:54.960677 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:54.964799 25101 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:54.985888 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:54.990630 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:54.991192 25023 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:54.991655 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:54.992642 25621 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20240426 13:20:54.992597 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2618) took 1210 ms. Trace:
I20240426 13:20:54.993129 25023 rpcz_store.cc:276] 0426 13:20:53.781992 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.991938 (+1209946us) service_pool.cc:225] Handling call
0426 13:20:54.992195 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.992335 (+   140us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:54.992358 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:54.992395 (+    37us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:54.992423 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.992470 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:54.992555 (+    85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.994438 25023 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:54.994892 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1854) took 1148 ms. Trace:
I20240426 13:20:54.995178 25023 rpcz_store.cc:276] 0426 13:20:53.846567 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.993916 (+1147349us) service_pool.cc:225] Handling call
0426 13:20:54.994169 (+   253us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.994702 (+   533us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.994849 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:54.979890 29511 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.997215 29511 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.998298 29017 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.987957 29005 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.999430 29017 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:54.999878 29005 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.000420 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:55.001731 29005 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.802587 29094 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.002220 29005 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:54.991573 29587 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.002647 29005 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.979717 29621 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.003367 29094 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.988073 29351 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:55.003803 29621 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.004303 29587 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.004508 29351 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.005785 29621 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:54.999186 29511 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.962607 29629 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:54.999167 29509 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:55.006605 29351 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.006961 29509 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:55.007726 29351 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:54.988976 25031 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:20:54.961534 28777 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.016414 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:54.985647 29359 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.017709 28777 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:54.987895 29218 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20240426 13:20:55.018061 29359 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.018678 25623 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20240426 13:20:55.007977 29629 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.950310 29421 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:55.007849 29509 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.991204 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:55.021754 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:55.022692 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:54.986896 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:55.014478 24952 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:20:54.823899 29571 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.023913 25025 raft_consensus.cc:2354] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:55.017565 29402 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:55.017517 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:54.981817 29557 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.159s	user 0.017s	sys 0.062s
I20240426 13:20:55.025027 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:55.025570 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:55.031167 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:55.031797 29447 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.019676 29628 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:54.959692 29374 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:54.986393 29388 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.034206 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:55.725067 25031 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:55.722769 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2462) took 2669 ms. Trace:
I20240426 13:20:55.727051 25025 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:20:55.727135 24952 rpcz_store.cc:276] 0426 13:20:53.053050 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.981511 (+1928461us) service_pool.cc:225] Handling call
0426 13:20:55.013817 (+ 32306us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:55.023775 (+  9958us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.722701 (+698926us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:55.085515 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59577.9us
I20240426 13:20:55.729568 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:55.577375 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.014s	sys 0.001s
W20240426 13:20:55.736125 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.019s	sys 0.002s
I20240426 13:20:55.730878 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:55.741393 25100 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:55.742570 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:55.730877 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 361113us
W20240426 13:20:55.587267 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 457251us
I20240426 13:20:55.725924 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1856) took 1742 ms. Trace:
I20240426 13:20:55.756188 25031 rpcz_store.cc:276] 0426 13:20:53.983835 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:55.015905 (+1032070us) service_pool.cc:225] Handling call
0426 13:20:55.724696 (+708791us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:55.725623 (+   927us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.725872 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:55.021071 29421 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:55.836659 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:55.562021 29220 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.001s
W20240426 13:20:55.842417 25105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.011s	sys 0.017s
I20240426 13:20:55.844878 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:55.270918 29629 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.032505 29639 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.052632 28777 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:55.034144 29624 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:55.851265 25105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.011s	sys 0.017s
I20240426 13:20:55.021934 29351 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:55.869360 29656 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.603s	user 0.000s	sys 0.000s
I20240426 13:20:55.870260 29351 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.870919 29351 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:55.484622 29359 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.446166 29058 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.367142 29588 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:55.849071 29220 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:20:55.820603 29633 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20240426 13:20:55.869247 29663 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:55.371804 29614 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.032385 29571 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.882632 29543 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:55.884377 29387 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.886049 29374 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:55.886907 29090 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.888197 29414 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:55.888346 29017 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.888s	user 0.007s	sys 0.003s
I20240426 13:20:55.912353 29663 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.925658 25618 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:55.926891 29663 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.858800 29629 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.929317 29629 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.844641 29490 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:55.929415 29571 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.097430 29621 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:55.914557 29218 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:55.871448 29351 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.930845 29218 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.932375 29351 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.910632 29370 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:55.933458 29370 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:55.933893 29370 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:55.905326 25619 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.925s	user 0.010s	sys 0.000s
I20240426 13:20:55.936383 25619 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:55.929630 29423 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:55.937628 29423 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:55.938129 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 798) took 2540 ms. Trace:
I20240426 13:20:55.938498 25622 rpcz_store.cc:276] 0426 13:20:53.397388 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.397597 (+   209us) service_pool.cc:225] Handling call
0426 13:20:53.819982 (+422385us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.820670 (+   688us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.937944 (+2117274us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400436550. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:55.938071 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":908375936}
I20240426 13:20:55.914041 29220 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
W20240426 13:20:55.876076 29557 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.002s	sys 0.025s
I20240426 13:20:55.945248 29009 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:55.945509 29557 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.003s	sys 0.025s
I20240426 13:20:55.904248 29534 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.874655 29624 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:55.912853 29543 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:55.948719 29543 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.949760 29543 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:55.950412 29503 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:55.952150 29634 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.395s	user 0.006s	sys 0.006s
I20240426 13:20:55.874917 29421 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.906841 29588 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.034355 29005 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:55.955451 29409 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:55.955786 29588 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.955839 29409 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:55.906600 29058 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.945259 29220 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:55.957173 29409 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.958007 29058 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.958236 29409 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:55.960431 29541 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 3.218s	user 0.013s	sys 0.010s
I20240426 13:20:55.960964 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 780) took 4992 ms. Trace:
I20240426 13:20:55.961351 25624 rpcz_store.cc:276] 0426 13:20:50.968694 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:50.968834 (+   140us) service_pool.cc:225] Handling call
0426 13:20:52.530486 (+1561652us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.108744 (+578258us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:54.136455 (+1027711us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:55.960766 (+1824311us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540056dbd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:55.960887 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4915005056}
I20240426 13:20:55.962599 29253 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:55.962982 29253 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:55.859943 29639 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.963783 29253 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.964994 29253 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:55.965662 29639 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:55.968083 29402 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.911s	user 0.006s	sys 0.003s
I20240426 13:20:55.968634 25617 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:55.930394 29621 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.969736 29388 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:55.970243 29618 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.152s	user 0.009s	sys 0.009s
W20240426 13:20:55.970445 29366 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.171s	user 0.012s	sys 0.000s
I20240426 13:20:55.970744 29481 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:55.971223 29481 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.971570 29621 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:55.957552 29220 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:55.973630 29220 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:55.974037 29481 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:55.975083 29573 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 3.225s	user 0.014s	sys 0.043s
W20240426 13:20:55.913060 29425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.017s	user 0.018s	sys 0.023s
W20240426 13:20:55.979610 29619 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.156s	user 0.015s	sys 0.018s
W20240426 13:20:55.949363 29630 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.124s	user 0.008s	sys 0.001s
I20240426 13:20:55.976815 29370 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.945115 29423 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:55.988732 29425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.095s	user 0.018s	sys 0.024s
I20240426 13:20:55.949882 29624 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:55.993650 29630 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.036895 29630 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:56.043118 29623 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.150s	user 0.013s	sys 0.023s
W20240426 13:20:55.842555 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.007s	sys 0.040s
I20240426 13:20:56.075949 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 770) took 5206 ms. Trace:
W20240426 13:20:56.076797 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.007s	sys 0.040s
I20240426 13:20:56.076844 24948 rpcz_store.cc:276] 0426 13:20:50.868953 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.464910 (+1595957us) service_pool.cc:225] Handling call
0426 13:20:52.465246 (+   336us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.767511 (+302265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.075046 (+3307535us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.075651 (+   605us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":586521984}
I20240426 13:20:53.484009 24947 rpcz_store.cc:276] 0426 13:20:51.232227 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.943382 (+1711155us) service_pool.cc:225] Handling call
0426 13:20:52.961236 (+ 17854us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.961487 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:53.225864 (+264377us) log.cc:842] Serialized 14 byte log entry
0426 13:20:53.456707 (+230843us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:53.456757 (+    50us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:53.456814 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.456888 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:53.458172 (+  1284us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27118,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":4228,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":14066,"thread_start_us":415045,"threads_started":2}
I20240426 13:20:56.079686 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.081336 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.079728 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.082324 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:56.106634 25022 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20240426 13:20:56.107337 25022 pending_rounds.cc:76] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 5
I20240426 13:20:56.107689 25022 pending_rounds.cc:97] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20240426 13:20:56.108208 25022 raft_consensus.cc:2866] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:56.111505 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2615) took 2420 ms. Trace:
I20240426 13:20:56.111948 25022 rpcz_store.cc:276] 0426 13:20:53.691026 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.713506 (+1022480us) service_pool.cc:225] Handling call
0426 13:20:56.105591 (+1392085us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.108947 (+  3356us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.111065 (+  2118us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400242550. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.111305 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3200210304}
I20240426 13:20:56.114240 25024 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:55.894780 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2594) took 4046 ms. Trace:
I20240426 13:20:56.117842 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2609) took 2938 ms. Trace:
I20240426 13:20:56.119382 25024 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:20:55.891454 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1822) took 4613 ms (client timeout 5371 ms). Trace:
W20240426 13:20:56.120246 25030 rpcz_store.cc:269] 0426 13:20:51.278214 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:51.398187 (+119973us) service_pool.cc:225] Handling call
0426 13:20:55.891197 (+4493010us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:55.891360 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4595368320}
I20240426 13:20:56.118319 25026 rpcz_store.cc:276] 0426 13:20:51.848676 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:52.488603 (+639927us) service_pool.cc:225] Handling call
0426 13:20:52.497106 (+  8503us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:52.767248 (+270142us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.894461 (+3127213us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54001e3554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:55.894688 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":513167616}
I20240426 13:20:56.117406 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2610) took 2442 ms. Trace:
I20240426 13:20:56.122043 25028 rpcz_store.cc:276] 0426 13:20:53.675201 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.716161 (+ 40960us) service_pool.cc:225] Handling call
0426 13:20:53.716664 (+   503us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.510240 (+793576us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.116731 (+1606491us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.117150 (+   419us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1548018432}
I20240426 13:20:56.082335 24947 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:20:56.089510 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:20:55.890372 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2228) took 2878 ms (client timeout 1662 ms). Trace:
W20240426 13:20:55.892907 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1809) took 2881 ms (client timeout 2045 ms). Trace:
W20240426 13:20:56.130843 25099 rpcz_store.cc:269] 0426 13:20:53.011843 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:55.892673 (+2880830us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:55.892808 (+   135us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240426 13:20:56.130724 25098 rpcz_store.cc:269] 0426 13:20:53.011610 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.016992 (+  5382us) service_pool.cc:225] Handling call
0426 13:20:55.890125 (+2873133us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:55.890303 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1380919680}
I20240426 13:20:56.085115 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.134017 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2245) took 1109 ms. Trace:
I20240426 13:20:56.134075 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.134514 25098 rpcz_store.cc:276] 0426 13:20:55.024224 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.132087 (+1107863us) service_pool.cc:225] Handling call
0426 13:20:56.132862 (+   775us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.133560 (+   698us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.133614 (+    54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.133678 (+    64us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:56.133716 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.133795 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.133909 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:56.090065 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2023) took 2756 ms (client timeout 2076 ms). Trace:
W20240426 13:20:56.094399 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2461) took 3043 ms (client timeout 1621 ms). Trace:
W20240426 13:20:56.136340 24949 rpcz_store.cc:269] 0426 13:20:53.050850 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.171389 (+120539us) service_pool.cc:225] Handling call
0426 13:20:56.093969 (+2922580us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.094214 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1430374144}
W20240426 13:20:56.136266 24951 rpcz_store.cc:269] 0426 13:20:53.333932 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.454554 (+120622us) service_pool.cc:225] Handling call
0426 13:20:56.089592 (+2635038us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540004c754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.089885 (+   293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310542720}
W20240426 13:20:56.138595 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1810) took 3126 ms (client timeout 2458 ms). Trace:
I20240426 13:20:56.139286 24951 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:20:56.139490 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2465) took 3002 ms. Trace:
I20240426 13:20:56.093343 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:56.140363 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 773) took 2584 ms. Trace:
W20240426 13:20:56.141261 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48239.7us
I20240426 13:20:56.142483 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:56.143522 25097 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:56.145289 24921 leader_election.cc:398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:56.145905 24921 leader_election.cc:401] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240426 13:20:56.118391 25027 rpcz_store.cc:276] 0426 13:20:53.178852 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.183430 (+  4578us) service_pool.cc:225] Handling call
0426 13:20:53.204858 (+ 21428us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:53.238733 (+ 33875us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:53.238782 (+    49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:53.256740 (+ 17958us) log.cc:842] Serialized 52 byte log entry
0426 13:20:53.426923 (+170183us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:53.427316 (+   393us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:53.427728 (+   412us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:55.978979 (+2551251us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:56.117068 (+138089us) raft_consensus.cc:1671] finished
0426 13:20:56.117113 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.117638 (+   525us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19523,"raft.queue_time_us":3440,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":13988,"spinlock_wait_cycles":70855040,"thread_start_us":112983,"threads_started":1,"wal-append.queue_time_us":476260}
I20240426 13:20:56.091589 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 778) took 4677 ms. Trace:
I20240426 13:20:56.092906 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 781) took 4128 ms. Trace:
I20240426 13:20:56.096609 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2457) took 3508 ms. Trace:
I20240426 13:20:56.168269 24955 rpcz_store.cc:276] 0426 13:20:51.964699 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.640876 (+1676177us) service_pool.cc:225] Handling call
0426 13:20:53.671259 (+ 30383us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.760172 (+ 88913us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.092326 (+2332154us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.092714 (+   388us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176075648}
I20240426 13:20:56.168350 24950 rpcz_store.cc:276] 0426 13:20:52.588454 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.861927 (+1273473us) service_pool.cc:225] Handling call
0426 13:20:53.862171 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:54.023843 (+161672us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.095963 (+2072120us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.096378 (+   415us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199018368}
I20240426 13:20:56.170233 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:56.167294 24954 rpcz_store.cc:276] 0426 13:20:51.414468 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.321753 (+1907285us) service_pool.cc:225] Handling call
0426 13:20:53.322113 (+   360us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:54.650931 (+1328818us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.090943 (+1440012us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.091404 (+   461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2759667968}
I20240426 13:20:56.171226 24950 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:56.086797 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 786) took 3362 ms. Trace:
I20240426 13:20:56.147732 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1309) took 1199 ms. Trace:
I20240426 13:20:56.172525 24954 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20240426 13:20:56.173521 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2019) took 2843 ms. Trace:
I20240426 13:20:56.172073 24953 rpcz_store.cc:276] 0426 13:20:52.723923 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.767392 (+2043469us) service_pool.cc:225] Handling call
0426 13:20:54.767771 (+   379us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:54.768117 (+   346us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:54.768151 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:54.768613 (+   462us) log.cc:842] Serialized 26 byte log entry
0426 13:20:55.078747 (+310134us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:55.079060 (+   313us) log.cc:842] Serialized 14 byte log entry
0426 13:20:55.079298 (+   238us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.079395 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:55.872688 (+793293us) spinlock_profiling.cc:243] Waited 96.9 ms on lock 0x7b180037d800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:56.086137 (+213449us) raft_consensus.cc:1671] finished
0426 13:20:56.086324 (+   187us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.086619 (+   295us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":103,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":1586,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":186126,"spinlock_wait_cycles":222844672,"thread_start_us":309768,"threads_started":1,"wal-append.queue_time_us":1135006}
I20240426 13:20:56.139963 24949 rpcz_store.cc:276] 0426 13:20:53.137554 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.138084 (+3000530us) service_pool.cc:225] Handling call
0426 13:20:56.138553 (+   469us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.139109 (+   556us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:56.139145 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.139185 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:56.139221 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.139290 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.139416 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.156625 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 788) took 3043 ms. Trace:
I20240426 13:20:56.156994 24947 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
W20240426 13:20:56.139073 25103 rpcz_store.cc:269] 0426 13:20:53.017463 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.074076 (+ 56613us) service_pool.cc:225] Handling call
0426 13:20:56.138155 (+3064079us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.138381 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106015104}
I20240426 13:20:56.140851 25096 rpcz_store.cc:276] 0426 13:20:53.555420 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.616482 (+ 61062us) service_pool.cc:225] Handling call
0426 13:20:53.616759 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.989498 (+372739us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.139841 (+2150343us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.140157 (+   316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684612992}
I20240426 13:20:56.148460 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.164204 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2227) took 3152 ms. Trace:
I20240426 13:20:56.170322 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:56.179626 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 789) took 2922 ms. Trace:
I20240426 13:20:56.175730 24953 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:56.173919 24954 rpcz_store.cc:276] 0426 13:20:53.329905 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.171658 (+2841753us) service_pool.cc:225] Handling call
0426 13:20:56.172200 (+   542us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:56.173166 (+   966us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.173449 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.180939 24951 rpcz_store.cc:276] 0426 13:20:53.112959 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.138029 (+3025070us) service_pool.cc:225] Handling call
0426 13:20:56.138708 (+   679us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.139890 (+  1182us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.156521 (+ 16631us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38656}
I20240426 13:20:56.183131 24951 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:56.183315 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:20:56.184074 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2467) took 2785 ms. Trace:
I20240426 13:20:56.180456 24947 rpcz_store.cc:276] 0426 13:20:53.257595 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.156333 (+2898738us) service_pool.cc:225] Handling call
0426 13:20:56.156736 (+   403us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.179403 (+ 22667us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.179580 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.179375 25097 rpcz_store.cc:276] 0426 13:20:53.011374 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.142522 (+3131148us) service_pool.cc:225] Handling call
0426 13:20:56.143008 (+   486us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.143924 (+   916us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.164137 (+ 20213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.172238 25102 rpcz_store.cc:276] 0426 13:20:54.947761 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.947878 (+   117us) service_pool.cc:225] Handling call
0426 13:20:56.145795 (+1197917us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.147436 (+  1641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139155840}
I20240426 13:20:56.157441 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.180102 24955 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:56.185667 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 849) took 1204 ms. Trace:
I20240426 13:20:56.185813 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:56.186128 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 845) took 2272 ms (client timeout 2869 ms). Trace:
I20240426 13:20:56.186664 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:56.188796 24921 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Timed out: Call waited in the queue past client deadline [suppressed 9 similar messages]
I20240426 13:20:56.176430 24949 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:56.206990 25617 rpcz_store.cc:269] 0426 13:20:53.913852 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.914358 (+   506us) service_pool.cc:225] Handling call
0426 13:20:56.185737 (+2271379us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.185916 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4554563456}
I20240426 13:20:56.208374 25618 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:56.194309 24954 rpcz_store.cc:276] 0426 13:20:53.398245 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.182321 (+2784076us) service_pool.cc:225] Handling call
0426 13:20:56.182819 (+   498us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.183821 (+  1002us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.184002 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.210021 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.183768 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2468) took 2785 ms. Trace:
I20240426 13:20:56.210719 24951 rpcz_store.cc:276] 0426 13:20:53.398552 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.182092 (+2783540us) service_pool.cc:225] Handling call
0426 13:20:56.182612 (+   520us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.183548 (+   936us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.183722 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.212309 24951 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20240426 13:20:56.212682 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.210738 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.214061 24947 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:56.214284 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.215202 25617 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:20:56.215710 24947 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20240426 13:20:56.207581 24952 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20240426 13:20:56.216431 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 792) took 2366 ms. Trace:
I20240426 13:20:56.189468 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2466) took 2836 ms. Trace:
I20240426 13:20:56.219794 24953 rpcz_store.cc:276] 0426 13:20:53.352599 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.174912 (+2822313us) service_pool.cc:225] Handling call
0426 13:20:56.175446 (+   534us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.189227 (+ 13781us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.189399 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.220257 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.220968 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.221311 25617 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:20:56.222496 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.223398 25617 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:56.191031 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.225625 25622 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:56.192319 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.192773 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.193325 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.189149 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.207755 25615 rpcz_store.cc:276] 0426 13:20:54.980988 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.981104 (+   116us) service_pool.cc:225] Handling call
0426 13:20:54.996318 (+ 15214us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:55.569739 (+573421us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:55.569788 (+    49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:55.585639 (+ 15851us) log.cc:842] Serialized 26 byte log entry
0426 13:20:55.912884 (+327245us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:55.912919 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:55.913014 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:56.185159 (+272145us) raft_consensus.cc:1671] finished
0426 13:20:56.185192 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.185458 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22677,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":582479,"threads_started":2,"wal-append.queue_time_us":371807}
I20240426 13:20:56.230130 24947 rpcz_store.cc:276] 0426 13:20:53.849647 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.214804 (+2365157us) service_pool.cc:225] Handling call
0426 13:20:56.215332 (+   528us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.216183 (+   851us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.216376 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.194672 24950 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:56.230538 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:56.218389 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:56.255187 25616 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:56.213047 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 791) took 2790 ms. Trace:
I20240426 13:20:56.257758 24951 rpcz_store.cc:276] 0426 13:20:53.422241 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.211495 (+2789254us) service_pool.cc:225] Handling call
0426 13:20:56.211913 (+   418us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.212794 (+   881us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.212989 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.259080 24951 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20240426 13:20:56.197212 29005 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.211426 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.233470 25024 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20240426 13:20:56.260663 25024 pending_rounds.cc:76] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 10
I20240426 13:20:56.261231 25024 pending_rounds.cc:97] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20240426 13:20:56.241598 29090 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.261531 25024 raft_consensus.cc:2866] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:56.261752 29090 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.262333 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 600) took 3022 ms. Trace:
I20240426 13:20:56.262753 25024 rpcz_store.cc:276] 0426 13:20:53.240005 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.267402 (+ 27397us) service_pool.cc:225] Handling call
0426 13:20:53.293404 (+ 26002us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.112961 (+2819557us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540028a8a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902adae1 00007fda9030e4b1 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2
0426 13:20:56.262076 (+149115us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.262270 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986621312}
I20240426 13:20:56.260735 29005 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.264544 29090 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.251228 29179 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.265239 29179 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.266072 29005 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.266613 29005 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:56.267310 29179 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.267381 29005 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.268321 29090 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:56.269042 29090 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.271011 29090 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.251919 29058 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:56.252801 25069 connection.cc:232] RPC call timeout handler was delayed by -1.7094889210000019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20240426 13:20:56.235774 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.272801 29058 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.273192 25069 connection.cc:232] RPC call timeout handler was delayed by -1.412923153999941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20240426 13:20:56.273663 25069 connection.cc:232] RPC call timeout handler was delayed by -1.412923153999941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20240426 13:20:56.274011 25069 connection.cc:232] RPC call timeout handler was delayed by -1.3985231539999177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
I20240426 13:20:56.274325 29366 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.253698 25621 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20240426 13:20:56.274860 25621 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 9
I20240426 13:20:56.275240 25621 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20240426 13:20:56.274904 29366 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.275396 29351 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.276430 29359 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:56.276620 29421 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:56.277222 29366 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.277084 29359 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.277748 29370 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.254330 29009 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.231349 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 176) took 29622 ms (client timeout 15000 ms). Trace:
I20240426 13:20:56.260844 24954 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:56.249675 29218 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:56.274431 25069 connection.cc:232] RPC call timeout handler was delayed by -1.3769231539998827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
I20240426 13:20:56.275565 25621 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:56.294883 29423 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:56.297740 25069 connection.cc:232] RPC call timeout handler was delayed by -1.1942231539999284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20240426 13:20:56.298233 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0106231539999726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
I20240426 13:20:56.298741 29429 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:56.279415 29388 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:56.299702 29429 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.300541 29120 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.301214 29120 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:56.298697 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0106231539999726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
I20240426 13:20:56.240936 24854 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 2 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:56.272068 25616 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:20:56.299978 29388 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:56.235158 25618 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:56.124887 24948 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:56.279633 29386 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:56.281137 29414 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:56.302297 29425 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:56.362470 29005 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.279282 29359 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.299219 29442 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:20:56.387291 29386 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:56.274997 29058 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.280591 29400 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:56.281430 29379 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:56.323493 24955 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:56.309583 29429 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.452615 24951 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20240426 13:20:56.450611 29562 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:56.409724 29425 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.457723 24954 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20240426 13:20:56.315305 29366 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.408607 29386 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:56.459172 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2027) took 1154 ms. Trace:
I20240426 13:20:56.284588 29351 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.458170 24954 pending_rounds.cc:76] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 7
I20240426 13:20:56.462921 24954 pending_rounds.cc:97] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20240426 13:20:56.463244 24954 raft_consensus.cc:2866] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:56.461696 24951 rpcz_store.cc:276] 0426 13:20:55.301539 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.259947 (+958408us) service_pool.cc:225] Handling call
0426 13:20:56.447750 (+187803us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.453100 (+  5350us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.456106 (+  3006us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54003c32d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.456302 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430780160}
I20240426 13:20:56.281826 29415 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.176s
I20240426 13:20:56.467916 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2028) took 1166 ms. Trace:
I20240426 13:20:56.468891 24954 rpcz_store.cc:276] 0426 13:20:55.301836 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.389919 (+1088083us) service_pool.cc:225] Handling call
0426 13:20:56.457262 (+ 67343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.467658 (+ 10396us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.467856 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.471057 29572 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.471851 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.003s	sys 0.007s
I20240426 13:20:56.471987 29425 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:56.472339 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.003s	sys 0.007s
I20240426 13:20:56.472882 29576 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.473491 29576 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.471740 29572 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.451622 29379 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:56.476240 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.003s	sys 0.007s
I20240426 13:20:56.445521 29009 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.485443 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.003s	sys 0.008s
I20240426 13:20:56.280149 29387 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.183s
W20240426 13:20:56.476143 25100 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.002s	sys 0.010s
W20240426 13:20:56.498152 25100 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.003s	sys 0.011s
I20240426 13:20:56.314415 29388 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:56.477366 29579 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.475776 29576 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.483062 29510 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.537545 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 127237us
I20240426 13:20:56.540731 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.483173 29379 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:20:56.550858 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 362078us
W20240426 13:20:56.521061 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.001s	sys 0.026s
I20240426 13:20:56.538192 29643 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.560189 24951 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20240426 13:20:56.542744 29551 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.566519 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.001s	sys 0.027s
I20240426 13:20:56.547060 29120 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.571359 25096 raft_consensus.cc:2396] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20240426 13:20:56.565137 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.575758 25098 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:56.575788 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:56.485595 29400 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.570103 29090 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.585829 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 328638us
I20240426 13:20:56.561813 29560 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:56.558209 29366 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.584978 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.593295 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2031) took 1140 ms. Trace:
I20240426 13:20:56.597529 24955 rpcz_store.cc:276] 0426 13:20:55.452981 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.455455 (+1002474us) service_pool.cc:225] Handling call
0426 13:20:56.465586 (+ 10131us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.592871 (+127285us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.592912 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.592966 (+    54us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:56.593013 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.593106 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.593234 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96185}
I20240426 13:20:56.577145 29677 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.602090 25105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.014s	sys 0.032s
W20240426 13:20:56.612653 29554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.543s	user 0.077s	sys 0.091s
I20240426 13:20:56.587016 29566 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:56.610152 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.516s	user 0.013s	sys 0.032s
W20240426 13:20:56.609947 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.019s	sys 0.024s
W20240426 13:20:56.602104 29521 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.028s	sys 0.028s
I20240426 13:20:56.609200 29503 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
W20240426 13:20:56.614852 25105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.014s	sys 0.032s
I20240426 13:20:56.568785 29579 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.549628 29510 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.639765 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.009s	sys 0.005s
W20240426 13:20:56.625190 29554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.624s	user 0.077s	sys 0.092s
W20240426 13:20:56.625624 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.600s	user 0.014s	sys 0.032s
W20240426 13:20:56.625929 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.020s	sys 0.025s
I20240426 13:20:56.627493 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:20:56.627985 29666 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.011s	sys 0.013s
I20240426 13:20:56.628253 29566 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.658478 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89260.6us
I20240426 13:20:56.658892 29538 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:56.630986 29446 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:56.660424 29446 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.662595 29446 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.664247 25098 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:56.664376 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57765.5us
W20240426 13:20:56.664914 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 848655us
I20240426 13:20:56.665143 29366 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.665324 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:56.666121 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.666293 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:56.666847 29677 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.613416 29090 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.632850 29351 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.668010 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:56.668666 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 607) took 3113 ms. Trace:
I20240426 13:20:56.669055 29677 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.670091 29400 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.670284 29683 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:20:56.670931 25097 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:20:56.671680 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 609) took 1709 ms. Trace:
I20240426 13:20:56.606462 29551 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.672156 25025 rpcz_store.cc:276] 0426 13:20:54.962485 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:55.806879 (+844394us) service_pool.cc:225] Handling call
0426 13:20:55.807302 (+   423us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:55.807586 (+   284us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:56.116324 (+308738us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.586667 (+470343us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.671117 (+ 84450us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.671180 (+    63us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:56.671289 (+   109us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.671387 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.671524 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93753,"prepare.queue_time_us":816658,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"thread_start_us":730097,"threads_started":2,"wal-append.queue_time_us":511896}
I20240426 13:20:56.673751 29551 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.673877 29644 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:56.674356 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2619) took 2751 ms. Trace:
I20240426 13:20:56.674769 25023 rpcz_store.cc:276] 0426 13:20:53.922885 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.995759 (+1072874us) service_pool.cc:225] Handling call
0426 13:20:54.995987 (+   228us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:55.014784 (+ 18797us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:55.025568 (+ 10784us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.673770 (+1648202us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.673954 (+   184us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.674009 (+    55us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:56.674088 (+    79us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.674178 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.674310 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10583,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"thread_start_us":1628888,"threads_started":1,"wal-append.queue_time_us":1610960}
I20240426 13:20:56.675947 29429 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:20:56.297428 24853 rpcz_store.cc:269] 0426 13:20:26.608814 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:26.609986 (+  1172us) service_pool.cc:225] Handling call
0426 13:20:56.231097 (+29621111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:48.482779 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:48.517401 (+ 34622us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:48.517467 (+    66us) write_op.cc:427] Acquired schema lock
0426 13:20:48.517480 (+    13us) tablet.cc:660] Decoding operations
0426 13:20:48.720614 (+203134us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:48.736758 (+ 16144us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:48.738677 (+  1919us) tablet.cc:683] Acquiring locks for 19 operations
0426 13:20:48.971362 (+232685us) tablet.cc:699] Row locks acquired
0426 13:20:48.971402 (+    40us) write_op.cc:260] PREPARE: finished
0426 13:20:48.971687 (+   285us) write_op.cc:270] Start()
0426 13:20:48.971853 (+   166us) write_op.cc:275] Timestamp: P: 1714137648971570 usec, L: 0
0426 13:20:48.971892 (+    39us) op_driver.cc:331] REPLICATION: starting
0426 13:20:48.972810 (+   918us) log.cc:842] Serialized 7378 byte log entry
0426 13:20:50.901506 (+1928696us) op_driver.cc:444] REPLICATION: finished
0426 13:20:51.367150 (+465644us) write_op.cc:300] APPLY: starting
0426 13:20:51.367274 (+   124us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:51.368944 (+  1670us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:51.369060 (+   116us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:52.369534 (+1000474us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:52.370077 (+   543us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=19,key_file_lookups=19,delta_file_lookups=19,mrs_lookups=38
0426 13:20:52.370155 (+    78us) write_op.cc:311] APPLY: finished
0426 13:20:52.370839 (+   684us) log.cc:842] Serialized 248 byte log entry
0426 13:20:52.371042 (+   203us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:52.371296 (+   254us) write_op.cc:446] Released schema lock
0426 13:20:52.371384 (+    88us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:52.377133 (+  5749us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:52.377177 (+    44us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":291633,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":455132,"apply.run_cpu_time_us":32717,"apply.run_wall_time_us":1036640,"cfile_cache_hit":36,"cfile_cache_hit_bytes":37638,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6218,"cfile_init":2,"lbm_read_time_us":153721,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":6,"num_ops":19,"prepare.queue_time_us":778808,"prepare.run_cpu_time_us":31207,"prepare.run_wall_time_us":1970313,"raft.queue_time_us":559145,"raft.run_cpu_time_us":51761,"raft.run_wall_time_us":375964,"replication_time_us":1920855,"thread_start_us":2528820,"threads_started":4,"wal-append.queue_time_us":1228402,"wal-append.run_cpu_time_us":16746,"wal-append.run_wall_time_us":3172847}]]}
I20240426 13:20:56.633867 29572 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.634387 29630 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:56.679407 29639 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:56.680171 29639 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:56.632530 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 902994us
W20240426 13:20:56.622340 29509 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.003s	sys 0.009s
W20240426 13:20:56.681691 29509 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.003s	sys 0.009s
I20240426 13:20:56.683146 29639 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.683595 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.612887 29555 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.636514 29379 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.637164 29562 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:56.628064 28777 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.000s	sys 0.014s
I20240426 13:20:56.640348 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.644095 29500 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.079s
I20240426 13:20:56.645416 29009 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.649032 24854 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:56.656862 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45879us
W20240426 13:20:56.609292 29550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.873s	user 0.038s	sys 0.091s
W20240426 13:20:56.656841 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.010s	sys 0.005s
I20240426 13:20:56.667559 29576 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:20:56.669137 29497 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:56.670497 29673 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:56.678314 29429 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:56.683950 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 780) took 1711 ms. Trace:
I20240426 13:20:56.683822 29678 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:56.684720 24956 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:56.684808 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:56.685228 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.686504 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.687256 29579 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.579252 29643 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.688004 29666 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.012s	sys 0.013s
I20240426 13:20:56.659726 29566 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.669098 25029 rpcz_store.cc:276] 0426 13:20:53.555633 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.611518 (+ 55885us) service_pool.cc:225] Handling call
0426 13:20:53.611767 (+   249us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:53.611958 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:53.677729 (+ 65771us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.131211 (+2453482us) spinlock_profiling.cc:243] Waited 6.8 ms on lock 0x7b18000701e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:20:56.668086 (+536875us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.668168 (+    82us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:56.668264 (+    96us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.668358 (+    94us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.668492 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52777,"prepare.queue_time_us":48419,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":72910,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":153,"spinlock_wait_cycles":15642496,"thread_start_us":409079,"threads_started":1,"wal-append.queue_time_us":2965522}
I20240426 13:20:56.622471 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.614331 29560 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:56.636161 29521 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.029s	sys 0.028s
I20240426 13:20:56.684805 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.691215 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:56.666628 29366 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:20:56.718237 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52206.8us
I20240426 13:20:56.702348 24955 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:56.700799 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.719358 25097 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:56.716645 25096 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
W20240426 13:20:56.699882 28777 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.000s	sys 0.015s
I20240426 13:20:56.695545 29576 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:56.711882 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84555.1us
W20240426 13:20:56.613052 29453 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.834s	user 0.040s	sys 0.053s
I20240426 13:20:56.696391 29497 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:56.699926 25105 rpcz_store.cc:276] 0426 13:20:54.972992 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.981533 (+  8541us) service_pool.cc:225] Handling call
0426 13:20:54.981801 (+   268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:55.012904 (+ 31103us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:55.024589 (+ 11685us) log.cc:842] Serialized 14 byte log entry
0426 13:20:55.906553 (+881964us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.636526 (+729973us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:56.636612 (+    86us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:56.636731 (+   119us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.683692 (+ 46961us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.683733 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.683890 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11792,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":7367,"raft.queue_time_us":57814,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":84,"spinlock_wait_cycles":70719872,"thread_start_us":1611047,"threads_started":2,"wal-append.queue_time_us":901801}
I20240426 13:20:56.697396 24951 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.6-13.6]
W20240426 13:20:56.698616 29550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.998s	user 0.039s	sys 0.097s
I20240426 13:20:56.726064 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.722083 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2032) took 1175 ms. Trace:
I20240426 13:20:56.727363 24948 rpcz_store.cc:276] 0426 13:20:55.542065 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.081854 (+539789us) service_pool.cc:225] Handling call
0426 13:20:56.717163 (+635309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:56.749084 25103 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:56.751571 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.713858 29677 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:56.718636 29366 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:56.753931 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.755038 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.755110 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.726191 29576 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:20:56.758546 24948 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:56.727106 29497 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:56.760650 25102 raft_consensus.cc:2396] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20240426 13:20:56.753101 29453 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.041s	user 0.040s	sys 0.053s
I20240426 13:20:56.758627 25105 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:56.767570 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.751188 29446 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:56.750205 29555 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.752781 29566 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:56.760210 24854 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 5 to 7, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 7 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:56.720798 28777 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:56.776124 29537 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:56.785252 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100592us
W20240426 13:20:56.786423 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102956us
I20240426 13:20:56.780239 29639 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:20:56.761197 29366 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:56.789602 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:56.780647 24948 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:56.782510 29446 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:56.794734 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.795665 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.796664 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.792294 29554 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.790463 29697 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.798462 29697 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.798369 29554 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.800884 29697 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.801965 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 796) took 1529 ms. Trace:
I20240426 13:20:56.802379 24947 rpcz_store.cc:276] 0426 13:20:55.272344 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.237821 (+965477us) service_pool.cc:225] Handling call
0426 13:20:56.238298 (+   477us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.238896 (+   598us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.238940 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.239615 (+   675us) log.cc:842] Serialized 26 byte log entry
0426 13:20:56.576937 (+337322us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:56.576978 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.577041 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:56.669738 (+ 92697us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:20:56.801651 (+131913us) raft_consensus.cc:1671] finished
0426 13:20:56.801678 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.801880 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":230966,"raft.run_cpu_time_us":2604,"raft.run_wall_time_us":191782,"spinlock_wait_cycles":8409344,"thread_start_us":334537,"threads_started":1,"wal-append.queue_time_us":467247}
W20240426 13:20:56.801033 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.000s	sys 0.033s
W20240426 13:20:56.804535 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.080s	user 0.000s	sys 0.038s
W20240426 13:20:56.807034 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45076.2us
I20240426 13:20:56.808355 24947 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240426 13:20:56.809566 25102 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:56.813772 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.815388 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.792361 29446 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:56.815997 25096 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:56.809124 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.817960 24948 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:20:56.818476 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.824777 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.826412 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:56.832147 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.822785 29630 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:20:56.836846 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2035) took 1133 ms. Trace:
W20240426 13:20:56.837682 24952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.010s	sys 0.018s
I20240426 13:20:56.838569 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:56.838850 24952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.010s	sys 0.018s
I20240426 13:20:56.839002 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.838764 24951 rpcz_store.cc:276] 0426 13:20:55.699941 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.575443 (+875502us) service_pool.cc:225] Handling call
0426 13:20:56.607498 (+ 32055us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.724523 (+117025us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.833108 (+108585us) spinlock_profiling.cc:243] Waited 88 ms on lock 0x7b54000a64d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.833353 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202436864}
I20240426 13:20:56.844597 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 787) took 3909 ms. Trace:
I20240426 13:20:56.840674 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.833762 29453 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:56.844918 24956 rpcz_store.cc:276] 0426 13:20:52.935688 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:54.959254 (+2023566us) service_pool.cc:225] Handling call
0426 13:20:54.959542 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.711890 (+1752348us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.836768 (+124878us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:56.844556 (+  7788us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3966608256}
I20240426 13:20:56.835446 29379 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:56.853575 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.81735e+06us
I20240426 13:20:56.851840 29557 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:56.857554 29557 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:56.857585 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:56.859388 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74947.9us
I20240426 13:20:56.859462 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.841320 29639 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.846104 29090 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.861749 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:56.850957 29453 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:56.831415 29537 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20240426 13:20:56.849750 29630 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:56.862445 25097 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:56.846186 29566 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20240426 13:20:56.875427 29653 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:56.872700 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:56.876749 29682 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:56.877132 29682 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:56.878000 29682 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.878587 29534 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:56.879431 29682 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:56.879289 29534 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.881197 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:56.851981 29370 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:56.824381 28777 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.857987 29555 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.864522 29503 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:56.867208 29675 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20240426 13:20:56.880930 29693 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20240426 13:20:56.863520 29677 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:56.852664 29386 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.896561 25621 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20240426 13:20:56.910138 25105 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:20:56.896768 29500 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.897195 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.897459 29630 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.853533 29351 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
W20240426 13:20:56.913978 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91022.8us
I20240426 13:20:56.901412 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.902413 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:56.903324 29695 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20240426 13:20:56.904110 29370 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.904441 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:56.891448 29579 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:56.906122 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:56.906809 29400 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:56.881834 29683 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20240426 13:20:56.857892 29660 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20240426 13:20:56.893226 29639 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.912459 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.910390 25621 pending_rounds.cc:76] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 7
I20240426 13:20:56.898681 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:56.932440 25621 pending_rounds.cc:97] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20240426 13:20:56.932849 25621 raft_consensus.cc:2866] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:56.913487 29503 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:56.913720 29555 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:56.928465 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:56.914382 29560 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.941890 24947 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20240426 13:20:56.948628 29379 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:56.949136 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.947726 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.964998 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:56.966785 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:56.967113 29554 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.968310 29557 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.969384 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 795) took 1697 ms. Trace:
I20240426 13:20:56.969695 29557 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:56.969993 24953 rpcz_store.cc:276] 0426 13:20:55.271975 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.222256 (+950281us) service_pool.cc:225] Handling call
0426 13:20:56.222697 (+   441us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.223230 (+   533us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.223274 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.223941 (+   667us) log.cc:842] Serialized 26 byte log entry
0426 13:20:56.606595 (+382654us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:56.606630 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.606717 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:56.969093 (+362376us) raft_consensus.cc:1671] finished
0426 13:20:56.969121 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.969307 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103851,"raft.queue_time_us":237932,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":20626,"thread_start_us":377064,"threads_started":1,"wal-append.queue_time_us":522295}
I20240426 13:20:56.972718 29534 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:56.972949 29537 consensus_queue.cc:1159] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 13 status { last_received { term: 13 index: 11 } last_committed_idx: 10 last_received_current_leader { term: 13 index: 11 } }
I20240426 13:20:56.974758 29688 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:56.975216 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.975873 29688 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:56.976297 29688 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:56.977246 29688 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.973414 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:20:56.979580 29688 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:56.980017 29643 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:56.980206 25101 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:56.981395 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:56.982044 29370 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.982620 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 797) took 1710 ms. Trace:
I20240426 13:20:56.983170 29645 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:56.983601 29645 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:56.946802 29453 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.984444 29453 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:56.984519 29645 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:56.985941 29645 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:56.990267 29678 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:56.926072 29677 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:56.979790 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.993361 29510 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:56.990633 29678 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:56.994038 29510 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:56.994840 29678 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:56.995597 29379 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:56.996063 29379 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:56.982993 24950 rpcz_store.cc:276] 0426 13:20:55.272638 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.255146 (+982508us) service_pool.cc:225] Handling call
0426 13:20:56.255865 (+   719us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.256354 (+   489us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.256376 (+    22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.256946 (+   570us) log.cc:842] Serialized 26 byte log entry
0426 13:20:56.575670 (+318724us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:56.672902 (+ 97232us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:20:56.816390 (+143488us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.821051 (+  4661us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.821119 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:56.982335 (+161216us) raft_consensus.cc:1671] finished
0426 13:20:56.982437 (+   102us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:56.982558 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4381,"prepare.queue_time_us":293581,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":219676,"raft.run_cpu_time_us":2378,"raft.run_wall_time_us":201565,"spinlock_wait_cycles":29261440,"thread_start_us":538828,"threads_started":2,"wal-append.queue_time_us":460750}
I20240426 13:20:56.996430 29379 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:56.996536 29639 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.997918 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:56.998977 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2034) took 1335 ms. Trace:
I20240426 13:20:56.999351 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.000347 29571 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:57.000619 29678 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:57.000754 29571 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:56.937803 29675 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20240426 13:20:56.956631 29388 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.001188 29571 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:56.947798 29009 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.036s
I20240426 13:20:56.999388 24954 rpcz_store.cc:276] 0426 13:20:55.663518 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.471178 (+807660us) service_pool.cc:225] Handling call
0426 13:20:56.973172 (+501994us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.978308 (+  5136us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.998784 (+ 20476us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b540004ae50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:56.998920 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1153537792}
I20240426 13:20:57.005429 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 790) took 3630 ms. Trace:
I20240426 13:20:56.937892 29555 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.015436 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:57.021744 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.023947 29120 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:57.024205 29551 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:57.025276 29551 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:56.940654 29500 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.025653 29551 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:57.021092 29555 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.026656 29551 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.027189 29500 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:56.948473 29503 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.030992 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 776) took 3173 ms. Trace:
I20240426 13:20:57.031879 29685 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.032397 29560 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:57.032444 29685 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.033131 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.033748 25103 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:57.033964 29560 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:57.035079 29685 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.031577 29370 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:57.037695 29629 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:57.038381 29629 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.038514 29387 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:57.038925 29629 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.040127 29562 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:57.040498 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73849.5us
W20240426 13:20:57.041136 29702 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20240426 13:20:57.042061 29666 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:57.042832 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2243) took 2031 ms. Trace:
I20240426 13:20:57.042196 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:57.030653 29366 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.046559 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:57.043327 25100 rpcz_store.cc:276] 0426 13:20:55.011558 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:55.834419 (+822861us) service_pool.cc:225] Handling call
0426 13:20:55.834761 (+   342us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.571522 (+736761us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.574326 (+  2804us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.601357 (+ 27031us) log.cc:842] Serialized 26 byte log entry
0426 13:20:56.847728 (+246371us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:56.847771 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.851158 (+  3387us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:57.042610 (+191452us) raft_consensus.cc:1671] finished
0426 13:20:57.042634 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:57.042778 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":99970,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":845385,"threads_started":2,"wal-append.queue_time_us":278289}
I20240426 13:20:56.961578 29179 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:57.006691 29521 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:57.016950 24949 rpcz_store.cc:276] 0426 13:20:53.374615 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.174993 (+2800378us) service_pool.cc:225] Handling call
0426 13:20:56.175554 (+   561us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:56.176195 (+   641us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:56.236998 (+ 60803us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.820619 (+583621us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.820923 (+   304us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:56.834950 (+ 14027us) log.cc:842] Serialized 52 byte log entry
0426 13:20:56.835046 (+    96us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:56.835067 (+    21us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.835135 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:57.003754 (+168619us) raft_consensus.cc:1671] finished
0426 13:20:57.003780 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:57.005325 (+  1545us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169688,"prepare.queue_time_us":274269,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":263399,"raft.run_cpu_time_us":428,"raft.run_wall_time_us":57041,"thread_start_us":537217,"threads_started":2,"wal-append.queue_time_us":470008}
I20240426 13:20:57.057005 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.048317 29683 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:57.071161 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.021513 29579 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:57.084944 25097 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:57.040784 29649 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:57.092311 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:57.087220 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.031380 25104 rpcz_store.cc:276] 0426 13:20:53.857102 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.857434 (+   332us) service_pool.cc:225] Handling call
0426 13:20:53.920665 (+ 63231us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:53.921010 (+   345us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:56.153396 (+2232386us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b180002f320. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:20:56.573235 (+419839us) log.cc:842] Serialized 14 byte log entry
0426 13:20:56.848359 (+275124us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.934034 (+ 85675us) log.cc:842] Serialized 26 byte log entry
0426 13:20:56.934198 (+   164us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:56.934242 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:56.953244 (+ 19002us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:56.962727 (+  9483us) raft_consensus.cc:1671] finished
0426 13:20:57.002965 (+ 40238us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:57.030830 (+ 27865us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7491,"prepare.queue_time_us":516806,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":10089,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":26874,"spinlock_wait_cycles":25948032,"thread_start_us":657851,"threads_started":2,"wal-append.queue_time_us":312587}
I20240426 13:20:57.050865 29120 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:57.096328 25100 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:57.044531 29666 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.012548 29576 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:57.111857 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 162866us
I20240426 13:20:57.053687 29688 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:57.121266 29563 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:57.122234 29560 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:57.122831 29560 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:57.123785 29560 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.078712 29388 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.055574 29521 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:57.091686 29551 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:57.130679 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 151019us
I20240426 13:20:57.132066 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.134104 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.137101 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.137449 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.137984 25103 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.084493 29683 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:57.136090 25104 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:57.138890 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:57.125108 29666 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.114939 29120 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.070801 29179 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.154974 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:57.157615 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94320.7us
I20240426 13:20:57.159435 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.132499 29649 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:57.162882 25618 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.052016 29677 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.164016 29660 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:20:57.143937 29683 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:57.175546 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:57.175546 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 793) took 1904 ms. Trace:
I20240426 13:20:57.145615 29386 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.153458 29120 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:57.180037 29628 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.132s
I20240426 13:20:57.185575 29630 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.192544 29521 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:57.193814 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93341.9us
I20240426 13:20:57.193892 29521 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:57.195885 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.176419 24952 rpcz_store.cc:276] 0426 13:20:55.271415 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.216663 (+945248us) service_pool.cc:225] Handling call
0426 13:20:56.217067 (+   404us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:56.217421 (+   354us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:20:56.839580 (+622159us) log.cc:842] Serialized 14 byte log entry
0426 13:20:57.175196 (+335616us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:57.175243 (+    47us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:57.175299 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:57.175380 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:57.175491 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24004,"prepare.queue_time_us":646207,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":37,"thread_start_us":882868,"threads_started":2}
I20240426 13:20:57.204492 25097 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:57.186064 29630 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:57.221441 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.177626 29550 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20240426 13:20:57.229420 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:57.227190 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.227052 29379 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.232218 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.227880 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.228902 24950 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:20:57.225067 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.224128 24949 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:20:57.208192 29387 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:57.252425 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.206295 29682 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20240426 13:20:57.254052 25023 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:20:57.255347 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:57.209759 29386 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:57.258873 29386 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.259295 29386 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:57.259176 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.260249 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.260892 24947 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:57.260993 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:57.227690 29707 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20240426 13:20:57.229632 29429 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20240426 13:20:57.235423 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.237648 29497 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.239272 29571 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.239785 29379 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.229532 29566 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20240426 13:20:57.213276 29500 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20240426 13:20:57.202739 29370 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20240426 13:20:57.280073 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.280623 25097 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 13
I20240426 13:20:57.280957 29550 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:57.281247 29497 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.281339 29550 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:57.240324 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.282724 24948 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:57.282424 29550 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.240895 29681 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20240426 13:20:57.284765 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.285357 24952 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:57.285594 29571 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.288787 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.289450 25622 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:57.223129 29366 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.244232 29675 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:57.290225 29675 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:57.283751 29550 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:57.247344 29579 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:57.291052 29579 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:57.240337 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82131.9us
W20240426 13:20:57.249965 29709 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:20:57.229368 29388 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20240426 13:20:57.262086 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.262640 24954 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:57.296626 25096 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:20:57.203035 29629 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.219224 29677 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.275779 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.276551 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.277338 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.279588 29429 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:57.279683 29566 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:20:57.290620 29675 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:57.186671 29649 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.246158 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.200455 29563 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.062s
I20240426 13:20:57.294926 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.320451 24950 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:57.206888 29446 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.298569 25618 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:57.305573 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:57.323933 25619 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:57.313180 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.325706 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.314216 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.327631 25024 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:57.315541 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.306921 29682 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20240426 13:20:57.328233 29555 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.330276 29555 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.316521 25624 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:57.345861 24954 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20240426 13:20:57.347481 24954 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:57.317441 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:57.313155 29429 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:57.361811 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.368988 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:20:57.362746 25620 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:20:57.367676 25097 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:57.400171 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.405596 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.425060 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.425844 25024 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:20:57.432847 29708 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:20:57.433600 29446 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.365712 29576 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.436504 29560 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:57.440337 29429 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.441584 29429 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:57.442746 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:57.442564 29120 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.443936 25619 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:57.444304 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:57.445955 29712 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20240426 13:20:57.446487 29120 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.448480 29555 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:57.452800 25100 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20240426 13:20:57.453205 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.453930 24949 raft_consensus.cc:2354] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:20:57.460326 29388 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:57.460940 29388 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:57.462014 29388 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.464641 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.465807 25025 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20240426 13:20:57.468741 29421 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:57.470153 29366 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:57.470780 29366 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.473695 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:57.474316 25619 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:57.366256 29386 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.475445 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:57.476008 25621 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:57.480484 29649 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.480847 29555 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:20:57.481654 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164268us
I20240426 13:20:57.482124 29555 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.484200 29688 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.484346 29120 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:57.484745 29688 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.489498 25097 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 13. Ops: []
I20240426 13:20:57.464818 29630 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:57.491286 25619 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:57.502504 25621 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:20:57.505290 25029 raft_consensus.cc:2354] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13.
I20240426 13:20:57.488953 25103 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:20:57.509399 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 103884us
I20240426 13:20:57.510206 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.510545 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.512702 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.513809 25103 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20240426 13:20:57.514957 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.521340 25023 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:57.523826 25621 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20240426 13:20:57.525202 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.531651 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.532585 25024 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:57.533164 25101 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20240426 13:20:57.536578 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.543773 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.526888 29572 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:57.556677 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:57.559628 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.564836 25023 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:57.542014 29566 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:57.570360 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.536676 29521 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.573793 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.557690 29572 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.537397 29534 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20240426 13:20:57.578115 25101 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:57.581147 25069 leader_election.cc:398] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:20:57.584187 25069 leader_election.cc:401] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:57.588568 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.572861 29566 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.594589 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.573381 29560 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:57.598062 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:57.621516 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.623306 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:57.628329 25101 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:57.590468 29572 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:57.603356 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20240426 13:20:57.600312 25025 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:57.627856 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:57.580066 29120 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:57.596469 29386 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.599017 29058 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:20:57.638161 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:57.644799 25100 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:20:57.577702 29521 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.638213 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.639619 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:57.646485 25620 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:20:57.602622 29566 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:57.590451 29550 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20240426 13:20:57.649271 25102 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:57.632187 29560 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.598642 29649 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20240426 13:20:57.636487 29568 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:57.647701 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.697547 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:57.708102 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 74627.1us
I20240426 13:20:57.708853 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:57.711118 29726 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20240426 13:20:57.712711 29717 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20240426 13:20:57.718079 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.718935 25103 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:20:57.724278 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.724709 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.656018 29688 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.663741 29120 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:57.692426 29560 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:57.762840 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:57.743613 29718 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20240426 13:20:57.744722 29719 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20240426 13:20:57.745501 29179 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:57.748262 29711 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:20:57.748205 25024 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
W20240426 13:20:57.749292 29724 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:20:57.749217 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.749660 29563 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:57.750039 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.750207 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.751358 25620 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
W20240426 13:20:57.751557 29720 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20240426 13:20:57.751885 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:57.794790 25025 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:57.753218 29678 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.752s	user 0.010s	sys 0.000s
I20240426 13:20:57.699570 29562 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20240426 13:20:57.809903 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.756441 29370 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:20:57.757685 29722 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20240426 13:20:57.735679 29675 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.765142 25023 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:57.734544 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45322.1us
I20240426 13:20:57.640985 29666 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.770044 25100 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:57.795614 25022 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:57.804863 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 53049.1us
I20240426 13:20:57.806876 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.718835 29579 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.815474 25024 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.790755 29550 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:57.835774 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.803862 29537 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:57.867866 25030 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:57.837787 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:57.871615 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.873092 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.869102 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:57.796785 29386 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:57.878353 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.878494 29386 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.879053 25022 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:57.815086 29563 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:57.879911 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.880523 29386 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.880746 25022 raft_consensus.cc:2396] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:20:57.878598 29537 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.885249 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:57.885890 29537 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:57.858412 29550 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.888355 29550 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:57.868134 29572 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.891515 29572 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:57.892410 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:57.894909 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:57.895810 29351 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:20:57.896659 29683 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.000s
I20240426 13:20:57.896792 25024 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:57.897111 29683 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:57.898007 29725 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20240426 13:20:57.899267 29683 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:57.814132 29370 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:57.901094 25096 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:57.875598 29678 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:57.914141 25027 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:57.901823 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.869582 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.897940 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.881497 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.906558 25622 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 14. Ops: []
I20240426 13:20:57.907641 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:57.921444 25022 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20240426 13:20:57.902323 29720 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:20:57.933187 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.926751 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.868410 29521 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:57.936630 29370 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:57.938273 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:57.901415 29179 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:57.907408 28777 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:20:57.940500 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58921.4us
W20240426 13:20:57.917403 29711 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:20:57.951989 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.952875 25025 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:57.953326 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.956650 28777 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:57.961825 28777 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:57.967566 25621 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:57.972098 25022 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:57.971019 25024 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:20:57.987387 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:57.988153 25024 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 15
I20240426 13:20:57.956096 29179 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:57.993295 29370 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:58.051595 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:57.998904 29179 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:58.059106 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.063302 29550 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:57.927733 29120 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.068955 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:58.071158 29742 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20240426 13:20:58.072207 29738 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20240426 13:20:58.072578 29735 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:58.074121 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75621.8us
I20240426 13:20:58.072072 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.130038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.135474 25621 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20240426 13:20:58.135833 25621 pending_rounds.cc:76] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 11
I20240426 13:20:58.136066 25621 pending_rounds.cc:97] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20240426 13:20:58.136292 25621 raft_consensus.cc:2866] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:58.141386 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:58.141940 25621 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:20:58.143839 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:58.144635 29566 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.145473 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:58.146688 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:57.983166 29571 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
W20240426 13:20:58.131961 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61900.8us
I20240426 13:20:58.129963 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.134291 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:58.147327 25022 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:58.151692 25031 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20240426 13:20:58.073863 29521 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:58.150074 25024 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:20:58.155853 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83856.8us
I20240426 13:20:58.158758 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.160552 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.171790 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.173929 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:58.151259 29743 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20240426 13:20:58.147245 29218 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:58.177100 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.183051 25022 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:20:58.069700 29415 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20240426 13:20:58.183998 25104 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.185024 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:58.171890 29734 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20240426 13:20:58.153838 29446 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:58.190224 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60355.4us
I20240426 13:20:58.191738 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:58.176086 29649 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:58.181953 29645 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:20:58.180956 29666 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:58.204596 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44115.7us
I20240426 13:20:58.205282 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.203426 29446 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:58.216192 29741 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20240426 13:20:58.217978 29740 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20240426 13:20:58.188000 29675 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.186568 29629 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:58.186654 29653 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:58.183846 29218 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:58.206185 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.225917 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.230690 25024 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:58.199196 29538 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:58.177579 29521 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.237550 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:58.248920 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:58.237250 24955 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:58.197687 29579 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:58.253865 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.250368 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.243675 25022 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:58.244450 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.227932 29566 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:58.202178 29649 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:58.255854 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.227783 29351 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:58.247807 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 99488.7us
I20240426 13:20:58.236465 29218 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:58.271595 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.282124 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 798) took 1386 ms. Trace:
I20240426 13:20:58.282471 24956 rpcz_store.cc:276] 0426 13:20:56.896203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.902155 (+  5952us) service_pool.cc:225] Handling call
0426 13:20:58.281011 (+1378856us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b54003d5ca0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda9031b08f 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:58.281935 (+   924us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:58.282056 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1419458304}
I20240426 13:20:58.283303 29560 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.267606 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.200990 29009 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.049s
I20240426 13:20:58.271230 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:58.269594 25101 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.638s	user 0.008s	sys 0.000s
I20240426 13:20:58.291466 25101 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:20:58.270277 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64187.6us
I20240426 13:20:58.248646 29388 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:58.286134 29747 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20240426 13:20:58.286581 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.286643 25621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:20:58.255420 25028 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:58.297935 25621 raft_consensus.cc:595] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.298307 25621 raft_consensus.cc:603] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:58.298517 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.289255 25024 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:58.289028 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.248469 29538 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:58.292053 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:58.351490 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54521.9us
I20240426 13:20:58.255061 28777 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.292729 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:58.254541 29562 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:58.254698 29645 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:58.354823 24953 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:58.236862 29653 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.353230 29538 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.356292 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:58.356839 29746 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20240426 13:20:58.359843 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.360630 24955 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:58.266319 29649 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:58.328395 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:58.330499 25105 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:58.295300 29351 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:58.348896 29550 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:58.373260 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81967us
I20240426 13:20:58.359408 29628 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:58.358894 29562 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:58.350289 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:58.352113 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60197.5us
I20240426 13:20:58.292685 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:20:58.352747 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.394270 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.395220 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.352304 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.395606 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.396795 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.351023 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.325261 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.397902 25097 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:58.398234 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2495) took 1053 ms. Trace:
I20240426 13:20:58.399514 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.376888 29179 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:58.378979 29706 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.230s	user 0.012s	sys 0.000s
I20240426 13:20:58.385406 29218 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:58.385352 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 433698us
I20240426 13:20:58.408551 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:58.389665 29576 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:58.389889 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:58.390292 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1862) took 1138 ms. Trace:
I20240426 13:20:58.411198 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.411810 25103 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:58.411502 25029 rpcz_store.cc:276] 0426 13:20:57.251909 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.252117 (+   208us) service_pool.cc:225] Handling call
0426 13:20:58.390113 (+1137996us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54001e4954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:58.390224 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536328960}
I20240426 13:20:58.397032 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:58.400739 29737 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:58.398591 24948 rpcz_store.cc:276] 0426 13:20:57.345286 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.352755 (+  7469us) service_pool.cc:225] Handling call
0426 13:20:57.403124 (+ 50369us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:57.509557 (+106433us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:57.550885 (+ 41328us) log.cc:842] Serialized 14 byte log entry
0426 13:20:57.638907 (+ 88022us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:58.384156 (+745249us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:20:58.384495 (+   339us) log.cc:842] Serialized 26 byte log entry
0426 13:20:58.384673 (+   178us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:58.384716 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:58.384791 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:58.391676 (+  6885us) raft_consensus.cc:1671] finished
0426 13:20:58.398056 (+  6380us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:58.398178 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20387,"prepare.queue_time_us":42345,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":5538,"raft.queue_time_us":27374,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":18564,"spinlock_wait_cycles":5160704,"thread_start_us":60441,"threads_started":1,"wal-append.queue_time_us":281927}
I20240426 13:20:58.404512 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.397903 29379 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:58.418709 25028 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:58.419514 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:58.405995 29754 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20240426 13:20:58.406384 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.420934 24949 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.407222 29521 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.404271 29644 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:58.414626 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.411330 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:58.417475 29744 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20240426 13:20:58.427349 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.428506 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.415339 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.392617 29628 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:58.399147 29562 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:58.407944 29009 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:20:58.413437 29497 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:20:58.416246 29662 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:58.409417 29645 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:58.416524 28777 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.417093 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.428202 29697 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:58.417690 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.418015 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.419988 25028 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:20:58.466959 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.411350 25100 raft_consensus.cc:595] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.291746 29387 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:58.430341 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.431058 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:58.434725 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:58.478158 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47919.9us
I20240426 13:20:58.478919 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.435290 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.474268 25029 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:20:58.424938 29673 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20240426 13:20:58.436836 24952 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20240426 13:20:58.419119 29576 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:58.437434 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2262) took 1553 ms. Trace:
I20240426 13:20:58.440304 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:58.423780 29644 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:58.417208 29179 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.443107 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.432078 29550 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:20:58.463601 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:58.462262 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.460747 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:58.487906 25098 rpcz_store.cc:276] 0426 13:20:56.884474 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.887446 (+  2972us) service_pool.cc:225] Handling call
0426 13:20:56.887739 (+   293us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:56.887908 (+   169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:56.887948 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:56.904959 (+ 17011us) log.cc:842] Serialized 26 byte log entry
0426 13:20:57.017338 (+112379us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:57.017371 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:57.017437 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:58.378407 (+1360970us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:58.437196 (+ 58789us) raft_consensus.cc:1671] finished
0426 13:20:58.437215 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:58.437383 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57407232,"thread_start_us":112040,"threads_started":1,"wal-append.queue_time_us":243908}
I20240426 13:20:58.500684 25617 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:58.461803 24954 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:58.463555 24948 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20240426 13:20:58.464504 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.438025 29662 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:58.429329 29653 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.471978 25103 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:58.479964 25618 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:20:58.436792 29628 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.435065 29629 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:58.483543 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.429968 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:58.485507 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.506510 25024 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:58.506654 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.483019 24949 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:20:58.436193 29386 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:58.488003 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.489187 25029 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240426 13:20:58.441026 29645 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:58.465209 29497 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:58.490979 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:58.516101 24952 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:58.516271 25028 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:58.472856 29697 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:58.472579 29560 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:58.479224 29387 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:58.472332 28777 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:58.482158 29571 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:58.529127 25618 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:20:58.508230 24950 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:58.492367 29644 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:58.492141 29351 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:58.492007 29673 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:58.512730 25025 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:20:58.510824 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.515153 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:58.515290 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94926.9us
I20240426 13:20:58.518311 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.518512 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.534365 24956 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:58.520399 25029 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:58.504359 29662 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:58.523622 24955 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20240426 13:20:58.524518 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45693.1us
I20240426 13:20:58.517846 29645 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.532618 25028 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:58.521639 29697 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.536242 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:58.522096 29386 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:58.526180 29387 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.543383 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.540143 29537 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:58.532119 28777 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:58.545328 29770 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20240426 13:20:58.535392 29673 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:58.551350 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 814) took 1271 ms. Trace:
I20240426 13:20:58.554308 29628 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.558267 29446 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.559664 29497 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.562747 24952 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:58.568635 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:58.561532 29351 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:58.570207 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.575505 25025 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [11.7-13.8]
I20240426 13:20:58.545559 29058 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:20:58.596313 25622 rpcz_store.cc:276] 0426 13:20:57.279814 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.290098 (+ 10284us) service_pool.cc:225] Handling call
0426 13:20:57.290354 (+   256us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:57.915596 (+625242us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:58.551167 (+635571us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:58.551302 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1416650624}
W20240426 13:20:58.603086 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71055.5us
W20240426 13:20:58.604210 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 93490.5us
I20240426 13:20:58.606751 29386 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.561879 29218 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.609583 25100 raft_consensus.cc:595] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.610250 24951 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:58.571247 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.621191 25100 raft_consensus.cc:603] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:20:58.580797 29370 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.000s
I20240426 13:20:58.577105 29660 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:58.624869 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:58.579339 29555 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:58.580677 29773 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:20:58.625950 24950 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:20:58.627976 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56907.9us
I20240426 13:20:58.629161 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:58.631937 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:58.635579 29673 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.639369 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:58.640087 29568 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:58.640515 29568 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:58.640882 29568 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:58.644557 29772 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:20:58.649041 29628 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.535384 29576 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.617183 29058 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:58.651713 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.686667 25025 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:58.677600 24954 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:20:58.689116 29555 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:58.690893 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:58.690238 29555 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.694936 29058 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.696764 29058 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:58.697345 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:20:58.700874 29776 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:20:58.701406 29776 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:58.702729 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.701442 29697 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.704314 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.702275 29776 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.706540 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.707437 29387 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.702047 29555 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.709756 29697 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.710901 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.713353 25025 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:58.678586 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53767.4us
I20240426 13:20:58.621995 29550 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:58.579715 29537 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.631071 28777 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:58.732893 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1826) took 2049 ms (client timeout 1694 ms). Trace:
W20240426 13:20:58.733353 25099 rpcz_store.cc:269] 0426 13:20:56.679820 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:56.714979 (+ 35159us) service_pool.cc:225] Handling call
0426 13:20:58.729372 (+2014393us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b540009c4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:58.729496 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1510425984}
W20240426 13:20:58.733821 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114704us
I20240426 13:20:58.650785 25100 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:20:58.650854 29534 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:58.634652 29775 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:20:58.676872 29773 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:58.737888 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:58.752910 24948 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:58.740906 25025 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:58.763303 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.738966 29566 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:58.741814 29534 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:58.753330 29387 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.763729 29678 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:58.769272 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.794632 29644 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.799129 29737 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:58.827209 29784 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.143s
W20240426 13:20:58.828263 29779 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20240426 13:20:58.840186 29752 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:20:58.790115 29534 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:58.764547 28777 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:58.787647 29566 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:58.836747 29773 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.753659 29351 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.850440 25025 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:58.846814 25616 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:20:58.855253 29566 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:58.857007 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1871) took 1310 ms (client timeout 1597 ms). Trace:
I20240426 13:20:58.862133 25024 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
W20240426 13:20:58.862440 25027 rpcz_store.cc:269] 0426 13:20:57.546374 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.546567 (+   193us) service_pool.cc:225] Handling call
0426 13:20:58.856777 (+1310210us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:58.856943 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":772710784}
W20240426 13:20:58.862989 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 172160us
I20240426 13:20:58.861016 25098 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
W20240426 13:20:58.865187 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 254991us
I20240426 13:20:58.791987 29649 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.823902 29678 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.880717 25617 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:58.885057 25027 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:58.884428 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.824337 29550 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.873946 29537 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.910678 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.907691 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.911844 25098 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.847703 29562 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:58.920133 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:58.905625 29538 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.550s	user 0.002s	sys 0.008s
I20240426 13:20:58.920787 25099 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:58.924491 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:58.925284 25097 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:58.921931 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:58.935554 29562 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.937140 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:58.931488 25098 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
W20240426 13:20:58.939389 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 203994us
I20240426 13:20:58.943279 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:58.944319 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.887231 29387 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20240426 13:20:58.889123 29682 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
W20240426 13:20:58.952189 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 343082us
I20240426 13:20:58.894645 29784 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20240426 13:20:58.955014 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:58.861778 28777 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.946614 29644 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.000296 29562 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:58.960358 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:58.975746 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.004442 29562 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:58.960042 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:59.007146 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.007848 29562 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:58.957806 24955 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:58.957141 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.000296 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.008514 25027 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 13
W20240426 13:20:59.010638 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53610.1us
I20240426 13:20:59.021432 25097 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:59.025486 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:58.938973 29538 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.034058 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:59.035269 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91028.2us
I20240426 13:20:59.038120 29662 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.039021 28777 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.040850 29662 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:58.957749 29683 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:59.053531 24952 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:59.042902 29576 raft_consensus.cc:2741] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20240426 13:20:59.061512 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:59.061390 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.062353 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.013070 29644 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:59.064975 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:20:59.056231 29379 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.628s	user 0.009s	sys 0.000s
I20240426 13:20:59.044986 29784 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:59.031625 29795 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:59.065865 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.093458 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:59.056411 29579 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:20:59.095093 29497 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:59.065862 29682 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:59.110033 25027 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:59.116163 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.092763 29379 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:59.069797 29683 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:20:59.117615 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.119298 25027 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:59.097815 29576 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:59.127306 24949 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:59.117362 29795 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:59.117431 29579 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.144934 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.113548 29644 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:59.124274 29537 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:59.106199 29179 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20240426 13:20:59.113624 28777 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:20:59.125507 29682 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.136991 29649 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.127637 29678 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.115731 29497 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:59.157301 29788 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20240426 13:20:59.181701 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.185480 25097 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:59.136627 29568 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.191669 25098 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:20:59.192811 25027 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:20:59.188726 29550 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:59.171046 29538 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.209383 24953 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:59.187374 29683 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.167184 29805 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20240426 13:20:59.218811 29538 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.006s
I20240426 13:20:59.218730 29644 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:59.225556 29810 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:20:59.125826 29784 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20240426 13:20:59.193969 29678 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20240426 13:20:59.255838 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109058us
W20240426 13:20:59.291911 29813 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20240426 13:20:59.296631 29787 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:59.297266 29787 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:59.226540 29799 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20240426 13:20:59.336197 29645 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.762s	user 0.007s	sys 0.004s
W20240426 13:20:59.336305 29798 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
W20240426 13:20:59.339293 29821 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20240426 13:20:59.316913 29568 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.353832 29538 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:59.219291 29579 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.232650 29682 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.346550 29645 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.346463 29534 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:59.372386 29806 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20240426 13:20:59.388193 25098 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20240426 13:20:59.398056 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91766us
I20240426 13:20:59.400158 25618 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:59.384096 29678 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:59.385313 29801 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:59.399487 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 285714us
I20240426 13:20:59.390607 28777 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:59.403627 29497 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.403710 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:20:59.405030 29817 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20240426 13:20:59.402101 29812 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20240426 13:20:59.405159 29534 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:59.402194 29643 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:59.406272 29366 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.407012 29645 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:59.408308 29576 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:20:59.408370 29762 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.049s	user 0.003s	sys 0.007s
I20240426 13:20:59.414424 29643 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:59.367801 29683 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.412243 25098 raft_consensus.cc:595] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.372200 29538 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.430692 25098 raft_consensus.cc:603] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:59.399363 29795 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.412074 29678 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.431834 25617 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:20:59.432307 29795 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.422267 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1886) took 1670 ms. Trace:
I20240426 13:20:59.435696 29678 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:59.424357 25621 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:20:59.413189 28777 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.428525 24956 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:59.439126 24954 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:59.442188 28777 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:59.443145 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.435474 25030 rpcz_store.cc:276] 0426 13:20:57.751395 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.926584 (+175189us) service_pool.cc:225] Handling call
0426 13:20:57.926976 (+   392us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:57.967956 (+ 40980us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:57.967983 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:57.978968 (+ 10985us) log.cc:842] Serialized 52 byte log entry
0426 13:20:58.248330 (+269362us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:58.264618 (+ 16288us) log.cc:842] Serialized 14 byte log entry
0426 13:20:58.295883 (+ 31265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:58.295953 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:59.407589 (+1111636us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:59.421952 (+ 14363us) raft_consensus.cc:1671] finished
0426 13:20:59.422051 (+    99us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:59.422213 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31048,"prepare.queue_time_us":15390,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":40604,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":8038,"spinlock_wait_cycles":7286144,"thread_start_us":117376,"threads_started":1,"wal-append.queue_time_us":245328}
I20240426 13:20:59.443789 24956 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:59.444729 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:59.445319 25098 raft_consensus.cc:595] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.414014 29801 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.446873 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:59.444727 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.447777 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:59.452116 28777 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:20:59.452610 25030 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:20:59.453200 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2642) took 1221 ms. Trace:
I20240426 13:20:59.453557 25022 rpcz_store.cc:276] 0426 13:20:58.231887 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.266139 (+ 34252us) service_pool.cc:225] Handling call
0426 13:20:59.453041 (+1186902us) spinlock_profiling.cc:243] Waited 84 ms on lock 0x7b5400242f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:59.453152 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":193117952}
I20240426 13:20:59.454026 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.454978 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.456248 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.453685 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.456751 24956 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:20:59.461275 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:20:59.462244 29801 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.460331 29538 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.467085 24956 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:20:59.473886 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.474795 24947 raft_consensus.cc:2354] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13.
I20240426 13:20:59.470662 29643 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.477283 29643 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:20:59.475227 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2043) took 1284 ms (client timeout 1622 ms). Trace:
I20240426 13:20:59.480839 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:59.480459 24947 rpcz_store.cc:269] 0426 13:20:58.190534 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.190669 (+   135us) service_pool.cc:225] Handling call
0426 13:20:59.473724 (+1283055us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:59.475169 (+  1445us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2515328}
I20240426 13:20:59.483623 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:59.484223 25098 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:20:59.485864 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.486917 29801 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:20:59.487286 29801 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:59.487624 29801 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:20:59.488772 29760 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.091s	user 0.004s	sys 0.014s
I20240426 13:20:59.503870 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 619) took 1637 ms. Trace:
I20240426 13:20:59.504256 25026 rpcz_store.cc:276] 0426 13:20:57.866582 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.943983 (+ 77401us) service_pool.cc:225] Handling call
0426 13:20:57.972897 (+ 28914us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:57.995557 (+ 22660us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:57.995584 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:57.995628 (+    44us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:20:57.995668 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:57.995721 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:59.503741 (+1508020us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5400415fd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:59.503824 (+    83us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26187008}
I20240426 13:20:59.510042 29058 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:59.510445 29058 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.511193 29058 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.512187 29058 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:59.516631 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 634170us
I20240426 13:20:59.522257 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.524245 29735 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:20:59.530354 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:59.533365 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.539865 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:59.542732 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:59.543061 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.544703 24953 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:59.546355 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.548306 29645 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:20:59.549480 29645 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:59.536005 29388 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.014s
I20240426 13:20:59.549842 29645 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:59.550729 29645 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.551923 29645 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:20:59.558532 29781 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.730s	user 0.005s	sys 0.004s
I20240426 13:20:59.555779 29801 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.564877 29628 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:20:59.566916 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.566373 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.570787 25103 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20240426 13:20:59.571131 25103 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 7
I20240426 13:20:59.571314 25103 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20240426 13:20:59.571487 25103 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:59.547080 29500 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20240426 13:20:59.575819 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.545274 29534 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20240426 13:20:59.584511 25103 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:20:59.587797 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.596441 29757 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240426 13:20:59.599342 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.22515e+06us
I20240426 13:20:59.599963 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:59.581491 29386 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:20:59.604228 29370 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:20:59.606848 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.611086 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.607729 29801 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:59.623919 25026 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:59.588739 29500 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:20:59.633105 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.638126 25621 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:59.638619 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.639493 29568 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:59.647161 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:59.614145 29386 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:59.648242 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:59.612990 29787 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:59.653635 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.636442 29534 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:59.644094 29735 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:59.643716 29120 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:20:59.637902 29500 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:59.651448 29386 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:59.662381 25103 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:20:59.632576 29683 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:20:59.656630 29801 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:20:59.654392 29503 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240426 13:20:59.650519 29415 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
W20240426 13:20:59.671473 24995 connection.cc:232] RPC call timeout handler was delayed by -1.4046674039999516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20240426 13:20:59.671921 24995 connection.cc:232] RPC call timeout handler was delayed by -1.197667403999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
W20240426 13:20:59.672233 24995 connection.cc:232] RPC call timeout handler was delayed by -1.1742674039999201s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
I20240426 13:20:59.675209 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 819) took 1055 ms. Trace:
I20240426 13:20:59.675556 24952 rpcz_store.cc:276] 0426 13:20:58.620134 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.620266 (+   132us) service_pool.cc:225] Handling call
0426 13:20:58.620541 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:59.054132 (+433591us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.675001 (+620869us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:59.675161 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":970484096}
I20240426 13:20:59.672850 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.679906 24952 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:20:59.683024 25100 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:20:59.684969 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:59.684083 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:59.651248 29568 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20240426 13:20:59.661083 29534 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:59.689929 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.698184 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2631) took 1471 ms. Trace:
I20240426 13:20:59.698500 29778 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.009s
I20240426 13:20:59.698560 29534 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.690475 29120 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:59.700171 29795 raft_consensus.cc:979] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:59.700646 29795 raft_consensus.cc:1067] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:20:59.700572 29120 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.698583 25024 rpcz_store.cc:276] 0426 13:20:58.227100 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.514534 (+287434us) service_pool.cc:225] Handling call
0426 13:20:58.856905 (+342371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:58.864460 (+  7555us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.698018 (+833558us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54003ae3d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:59.698134 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":786748160}
I20240426 13:20:59.664189 29735 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:59.702482 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2054) took 1213 ms. Trace:
I20240426 13:20:59.702575 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.703838 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.702488 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.704213 29429 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:59.704453 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.704685 29429 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:59.704893 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:20:59.705150 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.705505 29678 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:59.705716 29579 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:59.705914 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:59.706182 29678 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:20:59.687150 29822 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20240426 13:20:59.718866 25026 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:20:59.695926 29500 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.695417 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:59.690209 29218 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20240426 13:20:59.699195 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2511) took 1266 ms. Trace:
I20240426 13:20:59.705822 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:59.691452 25100 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:20:59.711349 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2655) took 1136 ms. Trace:
I20240426 13:20:59.720026 25103 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20240426 13:20:59.720083 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.703001 24951 rpcz_store.cc:276] 0426 13:20:58.489013 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.522040 (+ 33027us) service_pool.cc:225] Handling call
0426 13:20:58.610047 (+ 88007us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:58.622558 (+ 12511us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.702325 (+1079767us) spinlock_profiling.cc:243] Waited 79.3 ms on lock 0x7b54000a69d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:59.702432 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182446464}
I20240426 13:20:59.723601 24949 rpcz_store.cc:276] 0426 13:20:58.432738 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.514061 (+ 81323us) service_pool.cc:225] Handling call
0426 13:20:58.514314 (+   253us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:59.128244 (+613930us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.698957 (+570713us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:59.699144 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1405314944}
I20240426 13:20:59.724303 25026 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:59.724472 25027 rpcz_store.cc:276] 0426 13:20:58.575092 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.195345 (+620253us) service_pool.cc:225] Handling call
0426 13:20:59.196295 (+   950us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:59.610257 (+413962us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:59.610284 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:59.610633 (+   349us) log.cc:842] Serialized 26 byte log entry
0426 13:20:59.640190 (+ 29557us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:59.640232 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.640298 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:59.711109 (+ 70811us) raft_consensus.cc:1671] finished
0426 13:20:59.711136 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:59.711286 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":413563,"raft.queue_time_us":414015,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":29164,"threads_started":1,"wal-append.queue_time_us":29531}
I20240426 13:20:59.725528 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.726239 25024 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:59.726357 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.727048 25027 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:59.727114 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.728310 25024 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20240426 13:20:59.729099 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.730541 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:59.705070 29429 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:59.703375 29735 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.734387 25100 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:59.739249 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.745735 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:59.751443 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.719933 29678 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:59.752207 25024 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:20:59.752547 25100 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:59.721810 29675 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:59.732991 29560 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:20:59.734906 29653 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:20:59.731490 29090 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:59.737874 29379 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:59.749927 29837 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20240426 13:20:59.718704 29009 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20240426 13:20:59.740294 29759 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:59.775082 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.735659 29566 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.037s
I20240426 13:20:59.742965 29503 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:59.750434 29765 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:59.772358 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66603.4us
I20240426 13:20:59.773638 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.776360 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.787847 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:20:59.788925 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.754940 29500 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:20:59.731534 29555 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20240426 13:20:59.742977 29757 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:59.750411 29370 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20240426 13:20:59.757122 29678 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:59.769740 25026 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:20:59.784417 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45228.6us
I20240426 13:20:59.757834 29560 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:59.776937 29388 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:20:59.761322 25027 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:59.805826 25100 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:59.805140 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:59.890857 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.892138 25027 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:20:59.881363 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:59.771366 29769 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:20:59.767946 29366 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.041s
I20240426 13:20:59.803017 29555 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:20:59.795013 29563 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:20:59.894487 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89468.8us
I20240426 13:20:59.883975 29421 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:59.876971 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.901923 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:59.809026 29503 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:59.795182 29677 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.108s
I20240426 13:20:59.910540 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:59.922361 25618 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:59.807832 29675 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:20:59.912312 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:59.885594 29688 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:20:59.935274 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.784682 29379 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.792281 29759 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.795472 29653 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.107s
I20240426 13:20:59.913019 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:59.913662 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:59.921059 24948 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:59.921487 25620 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:59.795470 29763 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20240426 13:20:59.945318 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 170333us
I20240426 13:20:59.929524 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:20:59.896881 29666 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:20:59.932929 29685 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:20:59.932719 29503 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:59.799011 29767 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.113s
I20240426 13:20:59.796880 29757 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:59.910002 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2652) took 1388 ms. Trace:
I20240426 13:20:59.943465 25616 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:59.944125 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:59.796160 29765 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.008497 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.007469 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.028595 25620 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:20:59.953858 29415 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:20:59.954667 29763 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:00.014127 25026 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:59.941763 29677 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20240426 13:21:00.021808 29842 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20240426 13:21:00.024057 29753 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.735s	user 0.010s	sys 0.000s
I20240426 13:21:00.012401 29675 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.024317 29853 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:00.025241 24950 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:21:00.031457 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.013249 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:59.931355 29421 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.029870 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:00.040834 29572 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:00.041175 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97174.8us
I20240426 13:21:00.041713 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 887) took 1498 ms. Trace:
I20240426 13:21:00.042019 29677 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:00.042343 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2625) took 1816 ms (client timeout 2239 ms). Trace:
I20240426 13:21:00.010684 29685 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:00.057176 25024 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:00.079596 25031 rpcz_store.cc:269] 0426 13:20:58.225674 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.229863 (+  4189us) service_pool.cc:225] Handling call
0426 13:21:00.042200 (+1812337us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b54002ae254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:00.042295 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87866624}
I20240426 13:20:59.945308 29560 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.085697 29560 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:00.052947 29853 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:00.091568 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1897) took 1431 ms. Trace:
I20240426 13:21:00.057345 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:00.055377 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46957.9us
I20240426 13:21:00.055573 25027 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:00.097195 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.098049 25027 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:00.099273 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:00.091995 25022 rpcz_store.cc:276] 0426 13:20:58.660391 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.454441 (+794050us) service_pool.cc:225] Handling call
0426 13:20:59.454750 (+   309us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:59.611755 (+157005us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:59.618705 (+  6950us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.754402 (+135697us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:59.766311 (+ 11909us) log.cc:842] Serialized 26 byte log entry
0426 13:20:59.766456 (+   145us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:59.766523 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.766617 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.091333 (+324716us) raft_consensus.cc:1671] finished
0426 13:21:00.091356 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.091497 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":202549,"prepare.queue_time_us":6859,"raft.queue_time_us":176128,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":27399,"spinlock_wait_cycles":33560704,"thread_start_us":86085,"threads_started":1,"wal-append.queue_time_us":137292}
I20240426 13:21:00.103477 29677 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20240426 13:21:00.104593 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47368.2us
I20240426 13:21:00.105953 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.053500 29767 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:21:00.109138 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.110985 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.036784 29763 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:00.082163 29555 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:00.043282 29521 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20240426 13:21:00.118963 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.084282 25624 rpcz_store.cc:276] 0426 13:20:58.543346 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.544064 (+   718us) service_pool.cc:225] Handling call
0426 13:20:58.544345 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:58.749101 (+204756us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:20:58.953298 (+204197us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.014971 (+ 61673us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:59.015025 (+    54us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:59.015077 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:00.041436 (+1026359us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:00.041480 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.041594 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2154,"prepare.queue_time_us":101447,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":28981,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5261,"spinlock_wait_cycles":470333696,"thread_start_us":226307,"threads_started":2,"wal-append.queue_time_us":170269}
I20240426 13:21:00.139565 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.090831 29415 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:00.126214 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.126566 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:00.111755 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 234712us
I20240426 13:21:00.127599 29767 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:00.130630 25100 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:00.134685 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 617) took 2369 ms. Trace:
W20240426 13:21:00.188712 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49865us
I20240426 13:21:00.113714 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.106988 29666 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.172956 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.188197 25023 rpcz_store.cc:276] 0426 13:20:57.765116 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:57.931948 (+166832us) service_pool.cc:225] Handling call
0426 13:20:57.961300 (+ 29352us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:57.975562 (+ 14262us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:57.975602 (+    40us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:57.996338 (+ 20736us) log.cc:842] Serialized 78 byte log entry
0426 13:20:58.143654 (+147316us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:58.145237 (+  1583us) log.cc:842] Serialized 14 byte log entry
0426 13:20:58.160549 (+ 15312us) log.cc:842] Serialized 14 byte log entry
0426 13:20:58.180620 (+ 20071us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:58.180700 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:58.406911 (+226211us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b18003c1b40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:58.607255 (+200344us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b48000b19d8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:21:00.023327 (+1416072us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:00.134483 (+111156us) raft_consensus.cc:1671] finished
0426 13:21:00.134503 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.134635 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1040629,"prepare.queue_time_us":87503,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":7385,"raft.queue_time_us":241578,"raft.run_cpu_time_us":2486,"raft.run_wall_time_us":627384,"spinlock_wait_cycles":19279232,"thread_start_us":107762,"threads_started":4,"wal-append.queue_time_us":152508}
I20240426 13:21:00.172796 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.120164 29370 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:00.173847 25027 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:00.173861 24949 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:00.215106 25022 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:00.215373 24949 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:00.178491 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:00.190451 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:00.191176 25031 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:00.196414 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69941us
I20240426 13:21:00.219694 29218 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.138994 29555 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:00.143769 29853 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:00.224787 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 823) took 1794 ms (client timeout 1620 ms). Trace:
W20240426 13:21:00.225870 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2277) took 2068 ms (client timeout 1792 ms). Trace:
I20240426 13:21:00.226159 25624 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:00.226195 25105 rpcz_store.cc:269] 0426 13:20:58.157381 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.228949 (+ 71568us) service_pool.cc:225] Handling call
0426 13:21:00.225697 (+1996748us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:00.225815 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181918592}
W20240426 13:21:00.225160 25619 rpcz_store.cc:269] 0426 13:20:58.429916 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.430711 (+   795us) service_pool.cc:225] Handling call
0426 13:21:00.224583 (+1793872us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:00.224702 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314004224}
W20240426 13:21:00.230031 29838 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20240426 13:21:00.230341 29859 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.232218 29685 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.233455 29685 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:00.236335 29836 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.351s	user 0.001s	sys 0.000s
W20240426 13:21:00.240406 29846 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.301s	user 0.001s	sys 0.000s
I20240426 13:21:00.242290 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2648) took 1836 ms. Trace:
I20240426 13:21:00.250725 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.252867 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.254025 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.254997 24995 leader_election.cc:398] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13.
I20240426 13:21:00.255228 24995 leader_election.cc:401] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:00.186409 29677 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:00.258452 25023 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:00.142158 29769 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20240426 13:21:00.216558 29822 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:00.262923 29859 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:00.219290 29370 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:00.242653 25028 rpcz_store.cc:276] 0426 13:20:58.409996 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.568380 (+158384us) service_pool.cc:225] Handling call
0426 13:20:58.568659 (+   279us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:59.647397 (+1078738us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:59.647440 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:59.658552 (+ 11112us) log.cc:842] Serialized 52 byte log entry
0426 13:20:59.739570 (+ 81018us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:59.873579 (+134009us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.873767 (+   188us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.873885 (+   118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.242077 (+368192us) raft_consensus.cc:1671] finished
0426 13:21:00.242095 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.242234 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1082387,"prepare.queue_time_us":267517,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":1045019,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8833,"thread_start_us":180631,"threads_started":2,"wal-append.queue_time_us":128461}
I20240426 13:21:00.258356 24949 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:00.256628 29847 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
I20240426 13:21:00.221963 29666 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.280643 25028 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20240426 13:21:00.265879 29379 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:00.309911 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91365.8us
I20240426 13:21:00.279089 29677 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20240426 13:21:00.319337 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131086us
I20240426 13:21:00.275645 29370 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:00.314759 29768 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13.
I20240426 13:21:00.312671 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:00.353070 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.353988 29645 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:00.354457 29645 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:00.354840 29645 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:00.322206 29555 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:00.358260 29856 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20240426 13:21:00.359699 29844 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
W20240426 13:21:00.360375 29840 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.419s	user 0.002s	sys 0.000s
I20240426 13:21:00.360908 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1896) took 1700 ms. Trace:
I20240426 13:21:00.324929 29675 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.362993 29675 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.365320 29675 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.323647 29677 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:00.366549 29677 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:00.320715 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:00.369334 29865 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.361301 25030 rpcz_store.cc:276] 0426 13:20:58.660080 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.453237 (+793157us) service_pool.cc:225] Handling call
0426 13:20:59.453460 (+   223us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:59.611174 (+157714us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:59.611201 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:59.611574 (+   373us) log.cc:842] Serialized 26 byte log entry
0426 13:20:59.757580 (+146006us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:20:59.935202 (+177622us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.935377 (+   175us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.935451 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.360716 (+425265us) raft_consensus.cc:1671] finished
0426 13:21:00.360742 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.360856 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":190804,"prepare.queue_time_us":275737,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":165134,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":25230,"thread_start_us":296482,"threads_started":2,"wal-append.queue_time_us":297389}
I20240426 13:21:00.371752 25030 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:00.372421 29745 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:00.372750 29386 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.000s
I20240426 13:21:00.373952 29643 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:00.373150 29386 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.377224 24952 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:00.377646 29776 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:00.378010 29776 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:00.378727 29776 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.375597 29739 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240426 13:21:00.380571 29811 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.189s	user 0.001s	sys 0.008s
W20240426 13:21:00.344283 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72725us
W20240426 13:21:00.375533 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48386.6us
I20240426 13:21:00.295348 29859 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:00.309453 29534 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.320210 29864 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.084s
I20240426 13:21:00.383704 29534 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.384851 29566 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:00.385226 29566 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:00.385686 29666 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:00.385938 29566 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.387137 29566 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:00.386963 28777 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:00.390275 29653 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:00.390703 29653 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:00.391512 29653 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.392680 29653 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:00.393407 29572 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:00.395845 29388 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:00.296442 29415 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.351769 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:00.362341 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2646) took 1992 ms. Trace:
I20240426 13:21:00.287570 29759 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.400254 25029 rpcz_store.cc:276] 0426 13:20:58.370285 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.540726 (+170441us) service_pool.cc:225] Handling call
0426 13:20:58.542395 (+  1669us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:59.606538 (+1064143us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:59.606577 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:59.607112 (+   535us) log.cc:842] Serialized 26 byte log entry
0426 13:20:59.754035 (+146923us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:59.754068 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.754138 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.359683 (+605545us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:00.361403 (+  1720us) raft_consensus.cc:1671] finished
0426 13:21:00.361428 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.361920 (+   492us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1063725,"raft.queue_time_us":1064881,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":123879680,"thread_start_us":146573,"threads_started":1,"wal-append.queue_time_us":288888}
I20240426 13:21:00.402024 24949 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:00.402556 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50856.3us
I20240426 13:21:00.404278 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
W20240426 13:21:00.404805 29845 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.462s	user 0.001s	sys 0.000s
I20240426 13:21:00.404922 25622 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:00.406014 28777 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:00.406873 29859 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:00.018266 25025 rpcz_store.cc:276] 0426 13:20:58.521944 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:58.864961 (+343017us) service_pool.cc:225] Handling call
0426 13:20:58.884117 (+ 19156us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:58.914632 (+ 30515us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:59.105430 (+190798us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.888912 (+783482us) log.cc:842] Serialized 14 byte log entry
0426 13:20:59.908814 (+ 19902us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:59.909742 (+   928us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:59.909796 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:59.909856 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:59.909953 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":487258,"prepare.queue_time_us":533890,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":3804,"raft.queue_time_us":472848,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10299,"thread_start_us":489117,"threads_started":4,"wal-append.queue_time_us":108592}
I20240426 13:21:00.408203 29572 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:00.408779 29572 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:00.409430 29780 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.622s	user 0.010s	sys 0.000s
I20240426 13:21:00.409603 25030 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20240426 13:21:00.343989 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 116644us
I20240426 13:21:00.410735 29572 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:00.411268 29572 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:00.385994 29534 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.413327 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.410246 29645 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.413636 25022 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:00.415738 29864 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:00.415963 29645 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.348461 29563 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:00.417043 29628 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:00.417436 29563 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:00.417583 29628 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:00.418330 29563 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.419426 29563 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:00.418452 29628 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.319504 29765 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.420105 29628 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:00.372923 29745 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.423525 29745 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:00.427260 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43975.7us
I20240426 13:21:00.429737 25031 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:00.431890 25023 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20240426 13:21:00.281078 29678 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.433367 24949 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240426 13:21:00.349946 29429 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.000s
I20240426 13:21:00.444782 25022 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:00.446544 25022 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:00.450105 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 195216us
I20240426 13:21:00.451226 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.466017 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.466807 24956 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:00.477859 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:00.478456 25097 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:00.478787 25097 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:00.475548 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.489826 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:00.490890 25622 raft_consensus.cc:2396] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20240426 13:21:00.490420 25022 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:00.502735 25022 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:00.479631 25097 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.505723 25097 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:00.526806 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.461266 29745 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:00.558828 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:00.587359 24952 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:00.522711 29768 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:00.606328 25622 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:00.520375 29572 consensus_queue.cc:1159] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 11 status { last_received { term: 11 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 11 index: 7 } }
I20240426 13:21:00.463017 29429 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.477504 29765 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20240426 13:21:00.588372 29688 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:00.488679 29560 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.104s
W20240426 13:21:00.676818 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 79693.8us
I20240426 13:21:00.694203 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.696156 29058 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:00.696404 29866 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
I20240426 13:21:00.696774 29058 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:00.698158 29867 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:21:00.699427 29120 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:00.699849 29120 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:00.700683 29120 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.699203 29058 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:00.614043 25097 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:21:00.686149 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 122794us
I20240426 13:21:00.529543 29677 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.704411 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:00.596766 29500 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.801s	user 0.007s	sys 0.003s
I20240426 13:21:00.610180 29643 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:00.585923 29759 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20240426 13:21:00.745455 25028 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20240426 13:21:00.675585 29555 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:00.693536 29688 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:00.737880 29643 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:00.707856 29429 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:00.679267 29745 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.593816 29795 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.157s
I20240426 13:21:00.762567 29643 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:00.777930 29875 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20240426 13:21:00.778558 29822 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:00.781394 29784 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:00.781832 29784 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:00.783303 29685 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.550s	user 0.004s	sys 0.005s
I20240426 13:21:00.784451 29421 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.763197 29688 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.786986 29688 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.782733 29784 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.701764 29120 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:00.608757 29859 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:00.795792 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 124523us
I20240426 13:21:00.790786 29769 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.805277 29880 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:00.805650 29880 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:00.806375 29643 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.760044 29555 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.757547 29682 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20240426 13:21:00.809897 29555 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.810039 29643 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:00.811196 29763 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.000s
I20240426 13:21:00.812306 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1889) took 1066 ms. Trace:
I20240426 13:21:00.812726 25096 rpcz_store.cc:276] 0426 13:20:59.746172 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.746683 (+   511us) service_pool.cc:225] Handling call
0426 13:20:59.764974 (+ 18291us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:59.765157 (+   183us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:59.765184 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:00.366867 (+601683us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b2c0012e098. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:00.367176 (+   309us) log.cc:842] Serialized 26 byte log entry
0426 13:21:00.525606 (+158430us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:00.525627 (+    21us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:00.525669 (+    42us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.812065 (+286396us) raft_consensus.cc:1671] finished
0426 13:21:00.812083 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.812255 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6390656,"thread_start_us":158091,"threads_started":1,"wal-append.queue_time_us":425175}
I20240426 13:21:00.812258 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.815847 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2666) took 1204 ms. Trace:
I20240426 13:21:00.775933 29521 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:00.794359 29678 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:00.816490 25027 rpcz_store.cc:276] 0426 13:20:59.611656 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.215584 (+603928us) service_pool.cc:225] Handling call
0426 13:21:00.215959 (+   375us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:00.433963 (+218004us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:00.433992 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:00.441897 (+  7905us) log.cc:842] Serialized 26 byte log entry
0426 13:21:00.740353 (+298456us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:00.774576 (+ 34223us) log.cc:842] Serialized 14 byte log entry
0426 13:21:00.790549 (+ 15973us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:00.790648 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:00.815654 (+ 25006us) raft_consensus.cc:1671] finished
0426 13:21:00.815674 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:00.815782 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20923,"prepare.queue_time_us":34085,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":45808,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":194,"thread_start_us":431896,"threads_started":2,"wal-append.queue_time_us":328607}
I20240426 13:21:00.819648 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.821786 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.775502 29386 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.824407 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.806500 29880 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.827584 25105 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:00.836324 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.829586 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:00.837328 24953 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:00.838235 29745 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:00.839955 29677 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:00.830663 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.831004 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.831881 24950 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:21:00.832856 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.833737 24956 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20240426 13:21:00.844458 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.823961 29653 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:00.811601 29763 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.836987 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.835659 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.849565 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2667) took 1237 ms. Trace:
I20240426 13:21:00.857879 25026 rpcz_store.cc:276] 0426 13:20:59.611875 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.876652 (+264777us) service_pool.cc:225] Handling call
0426 13:21:00.849349 (+972697us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b5400056ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:00.849473 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213431168}
I20240426 13:21:00.833858 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.825496 25098 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:00.859787 24956 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:00.860327 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:00.854318 25096 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:00.861239 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:00.854885 25102 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:21:00.853505 24947 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:00.845922 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.871081 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:00.874485 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:00.848157 29784 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:00.863452 29677 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20240426 13:21:00.874462 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.878393 25026 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:00.874630 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:00.857697 29370 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.511s	user 0.008s	sys 0.000s
I20240426 13:21:00.886880 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:00.898012 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.898964 29677 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:00.868155 29848 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.767s	user 0.000s	sys 0.003s
I20240426 13:21:00.907842 25096 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:00.910986 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.919654 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.867640 29351 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.922044 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:00.916342 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 165409us
I20240426 13:21:00.922662 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:00.923421 25026 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:00.920717 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 183415us
I20240426 13:21:00.927979 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.929605 24956 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:00.937726 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.938567 24951 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:00.885130 29688 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:00.941792 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:00.947294 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 422980us
I20240426 13:21:00.884137 29218 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.954304 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:00.955432 24950 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:00.954357 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:00.957876 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:00.958065 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:00.874821 29839 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20240426 13:21:00.922070 29763 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:00.971146 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:00.933151 29682 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.977293 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:00.946532 29563 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.973217 25616 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:21:00.884407 29643 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20240426 13:21:00.973472 25624 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15.
I20240426 13:21:00.916813 29370 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:00.970579 29521 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:00.972106 29801 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:21:00.986562 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:00.995972 29683 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:00.996390 29768 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.007s
I20240426 13:21:00.999670 29877 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:01.000082 29871 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:01.000478 29868 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:01.000847 29862 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.000941 29677 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:00.996047 29765 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.001613 29822 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.004556 29757 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:01.009835 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:00.940481 29351 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.045482 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:01.046263 25031 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:01.047307 29351 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:01.023025 29370 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:01.023749 25099 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20240426 13:21:00.963738 29653 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20240426 13:21:01.027328 29768 consensus_queue.cc:1225] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:21:00.986845 29682 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:00.972662 29745 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20240426 13:21:01.051863 29682 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.052688 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.053941 25624 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:01.055230 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:00.973325 29421 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20240426 13:21:00.960088 29218 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.061164 29218 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:01.033351 24953 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
W20240426 13:21:01.033468 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45980.4us
I20240426 13:21:01.063967 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:01.038494 24950 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:21:01.047865 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 107019us
I20240426 13:21:01.055830 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:01.066884 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:00.979468 24954 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20240426 13:21:01.056916 29370 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:01.067566 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:01.020604 29683 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:01.017578 29801 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:01.071561 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.078912 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:01.075654 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:01.080736 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.073372 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:01.076444 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:01.083104 25098 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:01.035792 29892 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:21:01.084966 25026 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:00.948105 29688 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:01.081468 25097 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:00.993407 29563 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.094686 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:01.094094 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:01.098223 29560 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:01.098277 29905 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:01.089918 29682 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:21:01.100461 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67675.7us
I20240426 13:21:01.075533 29795 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:01.104629 29790 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:01.106823 29759 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:21:01.091861 29683 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:01.112787 25102 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:21:01.115746 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.117156 25621 raft_consensus.cc:1203] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 13.11->[15.12-15.12]   Dedup: 15.12->[]
W20240426 13:21:01.120023 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46741.5us
I20240426 13:21:01.123068 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:01.104197 29870 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:01.132946 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.136695 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.137224 25618 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20240426 13:21:01.141793 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:01.079078 29757 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:01.035679 29677 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.147508 25097 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:01.089881 29739 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.114588 29688 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:01.092661 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:01.129837 29685 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:01.156004 25621 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20240426 13:21:01.103861 29673 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:01.141376 29579 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:01.063114 29765 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.163900 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:01.164405 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.144928 29795 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.145509 29563 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:01.167299 25098 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:01.168016 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1910) took 1510 ms. Trace:
I20240426 13:21:01.146050 29759 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:01.147843 29560 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:01.116716 29745 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20240426 13:21:01.164490 24995 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15.
I20240426 13:21:01.183876 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:01.186270 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:01.186484 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:01.074601 29822 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.166651 29379 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20240426 13:21:01.215615 29908 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.211438 29853 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:01.218672 29819 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:01.221016 29819 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:01.043713 29862 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.223784 29862 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:01.158324 29503 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:01.224591 29503 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:01.224808 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.225411 29503 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.229281 25096 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:21:01.229578 29765 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.230422 29894 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.235817 25103 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:01.207571 29879 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.181780 29790 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:01.204663 24995 leader_election.cc:401] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20240426 13:21:01.178089 29675 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.218330 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:01.203086 25022 rpcz_store.cc:276] 0426 13:20:59.657870 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.528795 (+870925us) service_pool.cc:225] Handling call
0426 13:21:00.529164 (+   369us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:00.591305 (+ 62141us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:00.591333 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:00.672133 (+ 80800us) log.cc:842] Serialized 52 byte log entry
0426 13:21:00.919055 (+246922us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:00.957772 (+ 38717us) log.cc:842] Serialized 14 byte log entry
0426 13:21:00.960003 (+  2231us) log.cc:842] Serialized 14 byte log entry
0426 13:21:01.166688 (+206685us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:01.167117 (+   429us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:01.167369 (+   252us) raft_consensus.cc:1671] finished
0426 13:21:01.167394 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:01.167966 (+   572us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64379,"prepare.queue_time_us":25013,"prepare.run_wall_time_us":32,"raft.queue_time_us":117796,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":79938,"spinlock_wait_cycles":66008576,"thread_start_us":394280,"threads_started":2,"wal-append.queue_time_us":335267}
I20240426 13:21:01.188887 29643 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:01.170030 29739 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:01.245878 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 85411.1us
I20240426 13:21:01.247372 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:01.219751 29568 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:01.233383 25024 raft_consensus.cc:2396] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 13 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
W20240426 13:21:01.233561 29891 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20240426 13:21:01.197546 29795 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:01.243260 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:01.193403 25026 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:01.260749 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:01.261346 25026 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:01.243849 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:01.249583 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.262317 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:01.211393 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:01.188319 29757 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20240426 13:21:01.262243 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:01.264552 25026 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:01.219857 29908 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.241711 29685 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:01.236230 29894 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:01.292328 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81098.9us
I20240426 13:21:01.199326 29775 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20240426 13:21:01.270226 25098 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:01.240347 29853 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:01.294337 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:01.307736 25622 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:01.294767 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:01.298254 29908 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:01.251086 29745 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:01.300966 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:01.159680 29870 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.304937 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.300650 25031 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:01.251955 29568 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:01.332805 29568 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:01.264987 29767 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:01.267509 29643 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:01.310436 25026 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:01.260171 29120 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:01.221856 29819 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.254024 29879 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.249953 29675 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.268384 29759 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.318022 29894 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.268708 29660 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:01.298748 29415 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:01.431908 29415 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:01.432969 29675 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.432809 29415 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.437846 29870 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.293515 29370 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.452256 29370 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.454335 29370 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.455529 29822 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.456504 29677 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:01.460489 29643 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.345260 29790 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.312422 29560 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.320427 29521 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.321400 29853 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.477921 25617 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:01.478318 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.329116 29739 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:01.330785 29685 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.182277 29801 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.345295 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:01.379860 24920 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.4:40069 timed out after 1.754s (SENT) [suppressed 9 similar messages]
I20240426 13:21:01.406570 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.420085 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:01.309293 29868 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.430137 29887 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:01.431341 29897 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20240426 13:21:01.479713 29645 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.492403 25098 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:01.480100 29911 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.519985 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:01.428802 29660 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:01.495083 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 149855us
I20240426 13:21:01.502760 25026 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:01.481993 29820 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.543545 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:01.561607 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:01.488971 29521 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.538138 29908 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.568684 25098 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:21:01.510341 29579 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:01.555217 29660 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:01.548694 29859 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:01.555894 29773 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.556046 29677 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.555038 29218 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:01.511215 29683 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.575611 29579 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:01.722556 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:01.740685 25100 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:01.745820 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 365982us
I20240426 13:21:01.517747 29887 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.547209 29645 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.565621 29908 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.767743 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20240426 13:21:01.517335 29868 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.559219 29820 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:01.816973 29921 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.211s	user 0.000s	sys 0.000s
I20240426 13:21:01.741255 29579 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:01.883688 29521 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.784572 29218 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:01.931008 29218 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:01.936591 29918 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
W20240426 13:21:01.936782 29917 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
W20240426 13:21:01.938272 29919 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20240426 13:21:01.938979 29868 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.724123 29677 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.942242 29677 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.556804 29911 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:01.679564 29859 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.796643 29879 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:01.681550 29773 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.859555 29853 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:01.837700 29922 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:01.883770 29757 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:01.989097 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:02.007491 25100 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:02.008685 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.008304 29757 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.017395 29757 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.018065 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.018357 29879 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.019287 29879 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.020851 29876 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.021245 29876 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:01.994215 29868 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.029497 29868 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:02.029927 29920 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:02.031561 29868 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:02.032984 29568 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.700s	user 0.008s	sys 0.000s
I20240426 13:21:02.033071 29887 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.033430 29568 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.034850 29820 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.035185 29568 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.035926 29908 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.036123 29925 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:02.036586 29925 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.038365 29925 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:02.041486 29688 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.000s
I20240426 13:21:02.041867 29688 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.045874 29773 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.049556 29859 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.051591 29911 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.053098 29683 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.054523 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.055577 25104 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:02.005333 29906 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.057613 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:02.058904 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:21:02.056118 29120 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.622s	user 0.001s	sys 0.003s
I20240426 13:21:02.077113 25104 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20240426 13:21:02.077208 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.057190 29688 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.058902 29887 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.063486 29911 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.106122 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.106954 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:02.075008 29120 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15.
I20240426 13:21:02.053949 29579 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.008122 29922 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:02.110421 29795 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.814s	user 0.007s	sys 0.002s
I20240426 13:21:02.116704 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.116783 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.118592 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.121992 25104 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:02.120589 25022 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:02.124953 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.060556 29906 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.133788 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.134383 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.134652 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.138401 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.140725 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.141296 25620 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:02.142488 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.143191 25620 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:02.144362 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.145174 29579 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.145673 29922 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.145938 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.146046 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:02.146759 25620 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:02.147369 29660 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.540s	user 0.007s	sys 0.000s
I20240426 13:21:02.147610 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.147806 29660 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.148234 25025 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:02.105159 29887 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.148164 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.149307 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.149734 29579 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:02.150048 29887 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.150368 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:02.150236 29579 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.151041 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1922) took 1073 ms. Trace:
I20240426 13:21:02.151569 29887 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.127404 29795 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.151381 25025 rpcz_store.cc:276] 0426 13:21:01.077304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.150166 (+1072862us) service_pool.cc:225] Handling call
0426 13:21:02.150995 (+   829us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.159096 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.154567 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.158020 24947 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:21:02.156694 25097 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:02.160007 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1924) took 1067 ms. Trace:
I20240426 13:21:02.157162 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.162153 25025 rpcz_store.cc:276] 0426 13:21:01.092332 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.158820 (+1066488us) service_pool.cc:225] Handling call
0426 13:21:02.159964 (+  1144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.163152 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:02.109872 29911 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.155479 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.109045 29218 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.164918 24953 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20240426 13:21:02.167555 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 153037us
I20240426 13:21:02.217799 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.168063 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.175278 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.204200 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.214635 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:02.200379 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.271855 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.155248 29453 raft_consensus.cc:979] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:02.279783 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.287117 24956 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:02.301090 24951 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:02.306639 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1918) took 1589 ms. Trace:
I20240426 13:21:02.306965 24949 raft_consensus.cc:1203] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[7.4-7.4]   Dedup: 7.4->[]
I20240426 13:21:02.307129 25025 rpcz_store.cc:276] 0426 13:21:00.717334 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.167810 (+1450476us) service_pool.cc:225] Handling call
0426 13:21:02.306563 (+138753us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.307910 24949 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:21:02.308312 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:02.308665 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.311928 24953 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:21:02.313676 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.314437 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:02.315423 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.175381 29763 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:02.152046 29579 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.329674 25624 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:02.258745 29938 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.276777 29945 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.217140 29767 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.258695 29560 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:02.353868 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.341647 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.316339 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:02.217351 29906 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.304481 29877 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.260018 29922 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:02.354022 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50418.5us
I20240426 13:21:02.413059 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:02.357002 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.413906 25025 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:02.414366 24956 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:02.417215 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.219043 29660 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.359010 29795 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.379010 25104 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20240426 13:21:02.425688 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 117450us
I20240426 13:21:02.379809 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.326326 29822 raft_consensus.cc:1067] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:02.427155 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.152038 29887 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.403873 25098 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.354815 29120 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:02.403746 29914 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:02.357656 29763 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.403513 29937 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:21:02.432533 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 119758us
I20240426 13:21:02.416685 29922 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.380812 29767 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.445919 25025 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:02.453207 29956 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.453336 29951 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:02.416396 29560 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.429284 29685 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:02.434365 29908 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.380975 29945 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.418717 29877 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:02.454527 29935 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.297s	user 0.000s	sys 0.002s
I20240426 13:21:02.425297 29946 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:02.455410 24950 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:02.457708 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.457857 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.381076 29938 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.587288 29765 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.460767 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.614351 29782 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:02.615067 29782 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:02.615903 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 156425us
I20240426 13:21:02.623548 29945 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.448690 29120 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.624197 29945 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:02.625164 29945 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.625638 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1903) took 1555 ms. Trace:
I20240426 13:21:02.608521 29765 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.626415 29120 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.611500 24948 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:02.527089 25104 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:02.627589 29120 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.627751 29765 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.494194 29922 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.608512 29685 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.628898 29685 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:02.628132 29120 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:02.632876 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 637) took 1551 ms (client timeout 1895 ms). Trace:
W20240426 13:21:02.633287 25022 rpcz_store.cc:269] 0426 13:21:01.081630 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.243621 (+161991us) service_pool.cc:225] Handling call
0426 13:21:02.632717 (+1389096us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:02.632822 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954447744}
I20240426 13:21:02.634219 29945 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.634299 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:02.558760 29959 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20240426 13:21:02.616566 29782 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.635483 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:02.540189 29767 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.507562 29911 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.638403 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.641656 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:02.023782 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:02.645110 29879 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.643522 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:02.649825 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:02.651739 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.652350 25022 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:02.651757 24852 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:02.484730 29956 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.655535 29628 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.656306 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.656736 29956 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.657001 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:02.492560 29887 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:02.622547 29761 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.844s	user 0.015s	sys 0.010s
I20240426 13:21:02.657261 29956 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.658247 29966 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.664207 29956 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.494824 29218 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.570605 29560 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:02.622424 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 197727us
I20240426 13:21:02.450906 29763 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.626006 25103 rpcz_store.cc:276] 0426 13:21:01.070558 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.070683 (+   125us) service_pool.cc:225] Handling call
0426 13:21:01.070991 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:01.236179 (+165188us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.625414 (+1389235us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54000ea454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:02.625579 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378427776}
I20240426 13:21:02.678702 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.610137 29946 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.654444 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:02.713754 25022 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:02.654845 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:02.715644 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60870.8us
I20240426 13:21:02.631132 29120 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.621181 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.483282 29951 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.533304 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.664597 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.664382 29966 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.665072 29649 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:02.665762 29761 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.887s	user 0.016s	sys 0.012s
I20240426 13:21:02.762881 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.763731 25103 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
W20240426 13:21:02.659356 29878 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.925s	user 0.005s	sys 0.035s
I20240426 13:21:02.610981 29908 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.618341 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:02.659897 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 902) took 3037 ms (client timeout 1754 ms). Trace:
I20240426 13:21:02.668026 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:02.670702 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:02.770560 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102613us
I20240426 13:21:02.770823 25620 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20240426 13:21:02.617534 29938 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.619076 29877 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.656203 29628 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.777724 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.778520 25622 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20240426 13:21:02.779105 29628 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.773797 29739 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.768718 29673 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.780184 29955 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:02.780712 29955 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.780547 29673 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.781118 29955 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.774168 29649 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.782424 29673 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.771929 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:02.783264 25100 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.783696 29649 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.712735 29643 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.768810 29945 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:02.784325 29643 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.784487 29945 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:02.770037 25617 rpcz_store.cc:269] 0426 13:20:59.622071 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:59.622199 (+   128us) service_pool.cc:225] Handling call
0426 13:21:02.659745 (+3037546us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:02.659834 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4208567680}
I20240426 13:21:02.778980 29908 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.786222 29643 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:02.788892 29628 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.764506 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.769670 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:02.791071 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.791631 25025 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:02.678021 29218 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.794011 29218 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.762274 29946 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.776242 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.796926 25103 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:02.797451 29218 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.797741 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.786551 29961 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:02.798895 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.799125 29966 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.775265 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2670) took 3162 ms. Trace:
I20240426 13:21:02.761977 29951 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:02.800201 25023 rpcz_store.cc:276] 0426 13:20:59.612906 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.264685 (+651779us) service_pool.cc:225] Handling call
0426 13:21:00.431594 (+166909us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:00.432314 (+   720us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.775100 (+2342786us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400242a50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:02.775217 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345031040}
I20240426 13:21:02.779978 29739 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.780902 29761 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:02.806015 29761 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:02.806270 29739 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.806875 29761 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.807400 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.807919 29761 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.807976 25023 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:02.786222 29945 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.775295 29560 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:02.789727 25098 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:02.793351 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.792562 29628 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.772336 25022 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.798754 29961 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.775252 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.796254 25100 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:02.801476 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.803159 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.768908 29956 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.816288 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.816666 29956 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:02.816901 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:02.817041 29956 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.817407 29956 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.815177 29560 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.817981 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.772804 29120 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.795023 29685 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.809690 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:02.765053 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 310012us
I20240426 13:21:02.810880 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1892) took 2702 ms. Trace:
I20240426 13:21:02.821694 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.812393 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1900) took 1742 ms. Trace:
I20240426 13:21:02.822106 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:02.822320 25102 rpcz_store.cc:276] 0426 13:21:01.069536 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.070767 (+  1231us) service_pool.cc:225] Handling call
0426 13:21:01.070984 (+   217us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:01.150724 (+ 79740us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.808873 (+1658149us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:02.812330 (+  3457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95178496}
I20240426 13:21:02.820261 29560 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:02.823513 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.824896 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.825847 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.821839 25099 rpcz_store.cc:276] 0426 13:21:00.108549 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.110150 (+  1601us) service_pool.cc:225] Handling call
0426 13:21:00.137633 (+ 27483us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:01.048493 (+910860us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.810687 (+1762194us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:02.810838 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2037051776}
I20240426 13:21:02.813839 25025 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.798038 29946 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.827458 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.827474 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.815907 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:02.818665 25022 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:02.819039 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.819842 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.820163 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.820617 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.825549 29628 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.822335 25023 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.828377 25025 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:02.830919 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:02.826622 25102 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:21:02.833000 29645 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.831897 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.833169 29945 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.833902 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.834645 29872 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:02.834760 29787 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:02.835382 29888 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:02.835228 29787 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.835925 29888 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.836395 29888 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:02.836635 29787 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:02.835209 29872 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.826130 29120 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.838402 29805 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:02.838815 29805 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.833179 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.840514 25023 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:02.831831 29925 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:02.839413 29805 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:02.839578 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2540) took 2435 ms. Trace:
I20240426 13:21:02.845126 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:02.845530 24952 rpcz_store.cc:276] 0426 13:21:00.404051 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.405217 (+  1166us) service_pool.cc:225] Handling call
0426 13:21:00.405477 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:00.592536 (+187059us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.839354 (+2246818us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540004b854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:02.839528 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":371094912}
I20240426 13:21:02.846467 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.846714 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.838855 29872 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.829560 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.847000 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:02.847853 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.834794 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.849212 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.849493 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.841797 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:02.843781 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.844161 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.849298 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.841890 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.851281 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.851790 25022 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:02.851243 25617 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:02.852322 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1917) took 2135 ms. Trace:
I20240426 13:21:02.851140 25100 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:02.852499 24953 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20240426 13:21:02.852836 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.853101 25022 rpcz_store.cc:276] 0426 13:21:00.717054 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.841719 (+2124665us) service_pool.cc:225] Handling call
0426 13:21:02.852271 (+ 10552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.853830 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.849846 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.854720 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:02.855018 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:02.855496 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:02.854964 29782 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.861253 25025 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:02.839473 29865 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:02.861258 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:02.854516 25099 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:02.862048 25022 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:21:02.862533 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:02.853649 25622 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:02.860952 29643 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:21:02.864415 29643 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.864531 29662 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:02.845381 29388 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.864210 29782 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.847544 29925 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.862757 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 630) took 3121 ms. Trace:
I20240426 13:21:02.840945 29218 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:02.833781 29945 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.871905 25030 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:02.877429 29877 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:02.870391 29649 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:02.864931 29662 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:02.882035 29782 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.882045 29662 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:02.882529 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.883107 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2303) took 1751 ms. Trace:
I20240426 13:21:02.883491 25105 rpcz_store.cc:276] 0426 13:21:01.132140 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.132269 (+   129us) service_pool.cc:225] Handling call
0426 13:21:01.154936 (+ 22667us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:01.797529 (+642593us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.882870 (+1085341us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:02.883036 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1391369472}
I20240426 13:21:02.888104 29388 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.888753 29925 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.890105 29388 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.890667 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:02.890895 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.891379 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1912) took 3231 ms. Trace:
I20240426 13:21:02.891707 25030 rpcz_store.cc:276] 0426 13:20:59.660252 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.846172 (+3185920us) service_pool.cc:225] Handling call
0426 13:21:02.863043 (+ 16871us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.891133 (+ 28090us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.891328 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.840642 29739 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:02.845162 29769 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:02.871500 25022 rpcz_store.cc:276] 0426 13:20:59.741289 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.860167 (+3118878us) service_pool.cc:225] Handling call
0426 13:21:02.860437 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.862536 (+  2099us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.862709 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.842844 29120 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.856952 29628 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:02.893733 29628 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.894142 29628 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:02.894879 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 613) took 2306 ms. Trace:
I20240426 13:21:02.895087 25619 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:02.895279 25621 rpcz_store.cc:276] 0426 13:21:00.586731 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.589192 (+  2461us) service_pool.cc:225] Handling call
0426 13:21:01.117040 (+527848us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:01.188430 (+ 71390us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.892995 (+1704565us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b540053d9d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:02.893121 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1212982528}
I20240426 13:21:02.867879 29421 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:02.897471 29421 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:02.855978 29908 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:02.873189 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 624) took 3133 ms. Trace:
I20240426 13:21:02.872629 29946 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:02.861593 29645 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:02.912732 29645 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.913143 29645 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:02.915314 29946 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.915869 29946 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:02.898881 29961 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:02.900707 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.864881 29643 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:02.900641 25025 rpcz_store.cc:276] 0426 13:20:59.739442 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.854865 (+3115423us) service_pool.cc:225] Handling call
0426 13:21:02.855143 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.872904 (+ 17761us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.873095 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.899896 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.900969 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:02.868296 29865 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:02.902143 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.935091 25098 raft_consensus.cc:2396] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 13:21:02.902824 25099 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:02.903090 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:02.899451 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:02.878465 29218 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:02.936779 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:02.900424 29673 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:02.904412 29945 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:02.904225 29421 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.850107 29955 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:02.905194 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:02.988392 25105 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20240426 13:21:02.905778 29925 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:02.902307 29739 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:02.909773 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:02.989411 25617 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:02.936115 29908 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:02.929701 25030 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:02.992813 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 632) took 3251 ms. Trace:
I20240426 13:21:02.918216 29560 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.993139 25030 rpcz_store.cc:276] 0426 13:20:59.741755 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.892723 (+3150968us) service_pool.cc:225] Handling call
0426 13:21:02.929373 (+ 36650us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.992558 (+ 63185us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.992761 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:02.950363 29218 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:02.997941 29865 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.001727 29925 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:02.937428 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:02.936635 29782 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:02.949347 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1915) took 3005 ms. Trace:
I20240426 13:21:02.951606 29758 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:02.862684 29872 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:03.010082 25022 rpcz_store.cc:276] 0426 13:20:59.944284 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.896362 (+2952078us) service_pool.cc:225] Handling call
0426 13:21:02.896610 (+   248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.896795 (+   185us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:02.898522 (+  1727us) log.cc:842] Serialized 14 byte log entry
0426 13:21:02.948966 (+ 50444us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:02.949018 (+    52us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:02.949085 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:02.949162 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:02.949266 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":141,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":522,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":30162,"thread_start_us":47483,"threads_started":1}
I20240426 13:21:03.011900 24951 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20240426 13:21:03.002698 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66004.7us
I20240426 13:21:02.959062 29673 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:02.952844 29938 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:02.931999 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:21:03.015879 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83956.3us
I20240426 13:21:03.016579 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.008723 25025 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:03.019654 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.012365 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1911) took 3350 ms. Trace:
I20240426 13:21:03.012207 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.020641 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.011555 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:03.020922 25023 rpcz_store.cc:276] 0426 13:20:59.659023 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.845162 (+3186139us) service_pool.cc:225] Handling call
0426 13:21:02.911640 (+ 66478us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.911857 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:21:02.912175 (+   318us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.008751 (+ 96576us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.008801 (+    50us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:03.008852 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.008912 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.009015 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":197,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":79,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":3925,"thread_start_us":96260,"threads_started":1}
I20240426 13:21:03.013413 25621 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:03.008105 25617 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:21:03.020517 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.023310 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:03.018235 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.023895 25030 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:03.025946 29763 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.026311 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.027161 29966 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:03.027534 29966 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:03.027879 29966 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.011371 29758 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:03.021121 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:03.010026 29782 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:03.009672 29925 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.030951 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.033376 25622 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:03.052410 29945 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:03.052965 25025 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:21:03.022783 29769 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:03.054200 29685 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:03.073134 25030 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:21:03.053254 29951 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.014252 29872 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.069156 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2683) took 2974 ms. Trace:
I20240426 13:21:03.069104 25023 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 13:21:03.074270 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2681) took 3040 ms. Trace:
I20240426 13:21:03.021893 29908 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:03.074652 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2680) took 3048 ms. Trace:
I20240426 13:21:03.027993 29763 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.051412 29782 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:21:03.055083 29970 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20240426 13:21:03.082325 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.082882 25105 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.055994 29673 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.022994 29120 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20240426 13:21:03.051669 29877 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20240426 13:21:03.074692 25030 rpcz_store.cc:276] 0426 13:21:00.033777 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.024472 (+2990695us) service_pool.cc:225] Handling call
0426 13:21:03.072865 (+ 48393us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.074062 (+  1197us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.074233 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.072926 25022 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20240426 13:21:03.090667 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:03.076519 29872 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:03.073621 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.074398 25025 rpcz_store.cc:276] 0426 13:21:00.094864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.030905 (+2936041us) service_pool.cc:225] Handling call
0426 13:21:03.052616 (+ 21711us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.068946 (+ 16330us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.069116 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.032697 29956 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.099864 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:03.101369 29956 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.101760 29218 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.102437 29956 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:03.102864 29956 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:03.068686 29945 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.103663 29945 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.105187 29649 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:03.105782 25621 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:03.105916 29649 consensus_queue.cc:1225] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240426 13:21:03.076368 29739 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.106506 29649 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:03.106884 29649 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.065387 29784 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:21:03.033016 29938 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20240426 13:21:03.014853 29645 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.091315 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2682) took 3008 ms. Trace:
I20240426 13:21:03.103206 29956 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:03.120880 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.120955 25022 rpcz_store.cc:276] 0426 13:21:00.082640 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.030238 (+2947598us) service_pool.cc:225] Handling call
0426 13:21:03.072552 (+ 42314us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.091008 (+ 18456us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.091253 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.121845 25030 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.075253 25023 rpcz_store.cc:276] 0426 13:21:00.026404 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.022699 (+2996295us) service_pool.cc:225] Handling call
0426 13:21:03.065481 (+ 42782us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.074464 (+  8983us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.074609 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.103363 24952 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:03.028834 29961 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:03.016078 29944 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.055s
I20240426 13:21:03.077185 29908 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:03.110698 29560 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.070523 29628 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.128568 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.103801 29218 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:03.078312 29787 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.134985 25623 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:03.139055 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.139887 25022 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.141317 25616 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:03.088181 29662 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.152680 29628 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.126405 29955 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.128448 29784 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.126669 29961 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.129861 29908 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.164407 24953 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
W20240426 13:21:03.171047 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102492us
I20240426 13:21:03.174201 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.177587 25098 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.159308 29787 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.184080 25030 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:03.090140 29925 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.190219 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.190939 25030 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.193650 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.194198 25022 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:03.197070 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.197551 25022 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.200131 25023 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 14. Ops: []
I20240426 13:21:03.200757 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:03.201350 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 184429us
I20240426 13:21:03.178611 29961 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.202790 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.194723 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.204901 25623 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.209359 25098 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:03.209390 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.214172 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:03.214516 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.214951 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.215265 25099 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.215481 25030 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:03.210239 25105 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:03.217706 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.222226 25022 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:03.222996 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.223680 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.224872 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.226388 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.168807 29649 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.224404 29388 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:03.201684 29090 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:03.217877 29925 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.237558 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.243688 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.246245 29649 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:03.246484 29643 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.248669 29643 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.250187 29662 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.229801 29954 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:03.231552 29735 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:03.238814 29784 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.241843 29951 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.261395 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.263556 29645 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:03.264761 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.267386 25105 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:03.256155 29090 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.261912 29954 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.263104 29735 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.286058 25105 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:21:03.286908 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.288317 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:03.293998 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.294574 25030 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.295918 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.295696 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.278311 29761 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.299922 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 633) took 2688 ms. Trace:
I20240426 13:21:03.301014 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:03.301548 25029 rpcz_store.cc:276] 0426 13:21:00.611928 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.299018 (+2687090us) service_pool.cc:225] Handling call
0426 13:21:03.299317 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.299518 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:03.299557 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.299615 (+    58us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:03.299661 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.299731 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.299856 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:03.302961 29974 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20240426 13:21:03.301182 29909 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:03.316830 29954 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:03.311156 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 81853.9us
I20240426 13:21:03.311563 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.323235 25029 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:03.307413 25099 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:03.294150 29388 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.328742 25030 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:03.330117 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.292857 29945 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.333933 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.336504 25617 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:03.312534 29975 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:03.344517 29874 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.329404 29090 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.327582 29644 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:21:03.349534 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 640) took 2012 ms (client timeout 1824 ms). Trace:
W20240426 13:21:03.351915 25026 rpcz_store.cc:269] 0426 13:21:01.337194 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.419795 (+ 82601us) service_pool.cc:225] Handling call
0426 13:21:03.349376 (+1929581us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:03.349470 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22386432}
I20240426 13:21:03.353360 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1916) took 2636 ms. Trace:
I20240426 13:21:03.313028 29761 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.353730 25026 rpcz_store.cc:276] 0426 13:21:00.716745 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.352775 (+2636030us) service_pool.cc:225] Handling call
0426 13:21:03.353013 (+   238us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.353144 (+   131us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:03.353159 (+    15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.353186 (+    27us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:03.353209 (+    23us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.353240 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.353312 (+    72us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.360965 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:03.347642 29888 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.003s
W20240426 13:21:03.361876 25070 connection.cc:232] RPC call timeout handler was delayed by -1.0975635510000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20240426 13:21:03.370293 29758 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:03.371845 29915 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.056s	user 0.010s	sys 0.000s
I20240426 13:21:03.372418 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.378050 25029 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:03.374168 29388 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.378538 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:03.379060 25030 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 13. Ops: []
I20240426 13:21:03.378580 25099 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:03.379737 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.380311 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.382903 25029 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.381948 25623 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:03.347319 29980 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:03.400820 29980 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.400983 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.381362 29576 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:03.401712 29980 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.402303 29576 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.404716 29576 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:03.406694 29945 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.381948 29888 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.408975 29888 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.409683 24949 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:03.410288 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.410661 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.402453 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1906) took 2219 ms. Trace:
I20240426 13:21:03.385282 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:03.401834 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2698) took 2246 ms (client timeout 1771 ms). Trace:
W20240426 13:21:03.412163 25031 rpcz_store.cc:269] 0426 13:21:01.155859 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.242539 (+ 86680us) service_pool.cc:225] Handling call
0426 13:21:03.401635 (+2159096us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:03.401734 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93175040}
I20240426 13:21:03.412500 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:03.413292 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.414001 25031 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.414858 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.415417 25031 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.411327 25096 rpcz_store.cc:276] 0426 13:21:01.182736 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.193373 (+ 10637us) service_pool.cc:225] Handling call
0426 13:21:01.193716 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:01.229647 (+ 35931us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.402250 (+2172603us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b54001b4254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:03.402401 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80971520}
I20240426 13:21:03.384593 25617 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:03.384765 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.384874 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.385998 29218 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:03.388190 29370 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:03.388357 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2071) took 2443 ms. Trace:
I20240426 13:21:03.389902 29876 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.349347 29874 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.391036 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.392146 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:03.351361 29975 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:03.396270 29678 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:03.396735 29946 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.472736 29678 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:03.473166 29678 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.473590 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 618) took 1357 ms. Trace:
I20240426 13:21:03.473757 29946 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.397888 29908 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.473987 29678 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.475329 29678 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.475716 29908 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.441022 29370 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:03.476912 29370 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:03.325729 29805 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.478844 29956 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.479367 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.480146 29805 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.398608 29758 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:03.481521 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.481840 29956 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.482389 25096 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:03.452401 29888 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20240426 13:21:03.384280 29644 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:03.417415 29862 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:03.483577 29888 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:21:03.398893 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2685) took 3152 ms. Trace:
I20240426 13:21:03.417541 29784 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:03.421136 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.399958 29761 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.484587 25029 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.473995 25624 rpcz_store.cc:276] 0426 13:21:02.116135 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.116242 (+   107us) service_pool.cc:225] Handling call
0426 13:21:03.473432 (+1357190us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:03.473540 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443126016}
I20240426 13:21:03.420833 24955 rpcz_store.cc:276] 0426 13:21:00.944797 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.944966 (+   169us) service_pool.cc:225] Handling call
0426 13:21:00.945334 (+   368us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:00.965415 (+ 20081us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:00.965443 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:00.980072 (+ 14629us) log.cc:842] Serialized 26 byte log entry
0426 13:21:01.112823 (+132751us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b1800450820. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:01.256226 (+143403us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:01.256278 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:01.256363 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:03.371357 (+2114994us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:03.388105 (+ 16748us) raft_consensus.cc:1671] finished
0426 13:21:03.388133 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.388293 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13794,"raft.queue_time_us":31470,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":12463,"spinlock_wait_cycles":63902848,"thread_start_us":143204,"threads_started":1,"wal-append.queue_time_us":329883}
I20240426 13:21:03.480705 25022 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:03.464134 29967 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:03.464318 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.465067 29880 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.459856 25031 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.490885 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.467125 29872 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.491806 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.491925 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.492365 25031 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.492477 24955 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.467705 29563 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:03.468072 29975 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.468957 29218 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:03.341818 29735 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.481600 29874 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.482671 29576 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.020s
I20240426 13:21:03.483968 25620 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.483804 29644 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.394207 29782 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:03.462097 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77291.1us
I20240426 13:21:03.498405 29967 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:03.507372 25022 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:03.484264 25023 rpcz_store.cc:276] 0426 13:21:00.246077 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.128359 (+2882282us) service_pool.cc:225] Handling call
0426 13:21:03.196548 (+ 68189us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.203478 (+  6930us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.398706 (+195228us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b54003ae3d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:03.398820 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89409792}
I20240426 13:21:03.490435 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.468345 29453 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:03.497884 25029 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:03.498912 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:03.517331 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.499687 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:03.501224 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1919) took 2783 ms. Trace:
I20240426 13:21:03.501181 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.518122 25029 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:03.503023 25031 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:03.519013 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.515029 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.519670 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.520716 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.520932 29683 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:03.521739 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.521931 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.522432 24955 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:03.523213 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.524174 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.525043 25031 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:03.517701 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.525251 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.526085 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:03.526310 25029 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.483937 29888 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.502651 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.513975 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.527974 25621 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:03.529333 24995 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:03.529681 24995 leader_election.cc:401] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:03.531283 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.532423 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:03.533098 24953 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:03.534014 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.534673 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.535447 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2672) took 3922 ms. Trace:
I20240426 13:21:03.535943 25028 rpcz_store.cc:276] 0426 13:20:59.613471 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.297859 (+684388us) service_pool.cc:225] Handling call
0426 13:21:00.298218 (+   359us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:00.746006 (+447788us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.535117 (+2789111us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:03.535377 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1027897216}
I20240426 13:21:03.536638 25624 raft_consensus.cc:2354] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:21:03.537457 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.537535 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:03.540619 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.541956 24955 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:03.541890 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.543413 25029 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.544023 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.532078 25026 rpcz_store.cc:276] 0426 13:21:00.717599 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.358430 (+2640831us) service_pool.cc:225] Handling call
0426 13:21:03.358667 (+   237us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.358801 (+   134us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:03.362922 (+  4121us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.500922 (+138000us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.500981 (+    59us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:03.501026 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.501080 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.501179 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12124,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":10997,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":2545,"thread_start_us":128239,"threads_started":1}
I20240426 13:21:03.548174 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.548445 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.548812 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.550101 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:03.550237 25026 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.551299 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.547197 29366 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:03.498509 29058 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20240426 13:21:03.527704 29576 consensus_queue.cc:1225] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20240426 13:21:03.553512 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:03.557174 24995 leader_election.cc:401] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20240426 13:21:03.547319 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.557749 25023 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:03.557793 25029 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:03.503542 29784 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240426 13:21:03.528431 29862 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:03.559036 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.568439 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:03.560451 25029 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20240426 13:21:03.502133 29956 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20240426 13:21:03.562268 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2686) took 2844 ms. Trace:
I20240426 13:21:03.575423 25026 rpcz_store.cc:276] 0426 13:21:00.718143 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.549230 (+2831087us) service_pool.cc:225] Handling call
0426 13:21:03.549502 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.562041 (+ 12539us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.562217 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.575798 29956 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:03.576206 29956 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.576886 29872 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.576993 29956 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.577765 29888 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.578915 29888 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:03.579026 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.555043 28777 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:03.582489 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.583186 25096 raft_consensus.cc:2362] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14. Re-sending same reply.
W20240426 13:21:03.584444 29984 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:03.555555 29960 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:03.588761 29960 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:03.589169 29960 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.562208 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.535708 29683 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:03.573972 29366 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.574201 29967 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.574872 29782 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.579933 25105 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:03.593554 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2690) took 2824 ms. Trace:
I20240426 13:21:03.550511 29805 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:03.545817 29876 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:03.546545 29790 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:03.555155 29560 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:03.555387 29908 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.555357 29894 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:03.591571 29862 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:03.591234 29905 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:03.604236 25029 rpcz_store.cc:276] 0426 13:21:00.769311 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.559755 (+2790444us) service_pool.cc:225] Handling call
0426 13:21:03.560156 (+   401us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:03.593241 (+ 33085us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.593490 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.640738 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:03.636893 25027 raft_consensus.cc:2396] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20240426 13:21:03.599532 29683 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:03.631031 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79784.6us
I20240426 13:21:03.591039 29576 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:03.641366 25029 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:03.605935 29090 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:03.614203 29876 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:03.658295 29876 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.645871 29683 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:03.648356 29366 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.602305 29218 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.650029 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:03.646607 29859 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:03.614900 29560 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:03.654274 25105 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:03.656515 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2687) took 2938 ms. Trace:
I20240426 13:21:03.681105 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:03.606810 29956 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.636974 29805 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:03.658607 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:03.640439 29862 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.660362 24955 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.640849 29790 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:03.655030 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 627) took 3914 ms (client timeout 2802 ms). Trace:
W20240426 13:21:03.735054 25024 rpcz_store.cc:269] 0426 13:20:59.740558 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.127753 (+387195us) service_pool.cc:225] Handling call
0426 13:21:03.649694 (+3521941us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:03.654966 (+  5272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2345096576}
W20240426 13:21:03.655977 29966 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.628s	user 0.006s	sys 0.004s
I20240426 13:21:03.668694 25029 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:03.736780 25029 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:21:03.669631 29370 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.674548 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2322) took 1046 ms. Trace:
W20240426 13:21:03.676770 29982 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:21:03.677532 25098 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:21:03.678740 24953 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:21:03.678740 24951 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.680308 25028 rpcz_store.cc:276] 0426 13:21:00.718398 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.549659 (+2831261us) service_pool.cc:225] Handling call
0426 13:21:03.550971 (+  1312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.551144 (+   173us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:21:03.564574 (+ 13430us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.656143 (+ 91569us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.656205 (+    62us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:03.656264 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.656332 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:03.656456 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39643,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":79,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":3942,"thread_start_us":90978,"threads_started":1}
I20240426 13:21:03.662647 29090 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:03.724728 29666 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:03.689513 29560 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:03.737175 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1921) took 2668 ms. Trace:
I20240426 13:21:03.757417 25029 rpcz_store.cc:276] 0426 13:21:01.068419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.736270 (+2667851us) service_pool.cc:225] Handling call
0426 13:21:03.736548 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.737021 (+   473us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.737142 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.737169 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:03.758323 25029 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:03.758754 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1925) took 2663 ms. Trace:
I20240426 13:21:03.758963 25029 rpcz_store.cc:276] 0426 13:21:01.095705 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.757989 (+2662284us) service_pool.cc:225] Handling call
0426 13:21:03.758214 (+   225us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.758603 (+   389us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.758720 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.739053 25104 rpcz_store.cc:276] 0426 13:21:02.628482 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.664378 (+ 35896us) service_pool.cc:225] Handling call
0426 13:21:03.674246 (+1009868us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b540017f628. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda96290034 00007fda9525a0a7 00007fda95259838 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:03.674502 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2491264}
I20240426 13:21:03.759887 25029 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:03.760759 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2697) took 2624 ms. Trace:
W20240426 13:21:03.696879 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46975.7us
I20240426 13:21:03.761555 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:03.760972 29090 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.762804 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:03.764115 29453 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:03.764179 29566 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:03.764997 29566 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:03.765290 29566 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.749136 25024 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20240426 13:21:03.725865 29745 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:03.766445 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2695) took 2948 ms. Trace:
I20240426 13:21:03.766043 29566 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.766952 25024 rpcz_store.cc:276] 0426 13:21:00.818095 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.735726 (+2917631us) service_pool.cc:225] Handling call
0426 13:21:03.748932 (+ 13206us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:03.766126 (+ 17194us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.766381 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.767444 29566 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:03.661048 29576 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.742899 29643 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:03.726771 29872 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:03.738901 29989 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:03.768636 25029 rpcz_store.cc:276] 0426 13:21:01.136368 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.759441 (+2623073us) service_pool.cc:225] Handling call
0426 13:21:03.759684 (+   243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.760533 (+   849us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.760715 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.776404 25024 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:03.738152 29944 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:03.781234 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.776988 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1927) took 2597 ms. Trace:
I20240426 13:21:03.782045 25024 rpcz_store.cc:276] 0426 13:21:01.179135 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.767998 (+2588863us) service_pool.cc:225] Handling call
0426 13:21:03.768637 (+   639us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.776777 (+  8140us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.776943 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.777480 29370 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:03.770409 29643 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.768779 29576 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.782490 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:03.776667 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:03.811558 29179 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:03.813498 25029 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:03.814163 29179 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.815248 25029 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.783244 24951 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:03.816982 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.819079 25023 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:03.757308 29058 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20240426 13:21:03.803033 29805 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:03.820256 29678 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:21:03.820834 29058 consensus_queue.cc:1225] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:21:03.821328 29058 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:03.821401 29678 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:03.821693 29058 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.822024 29678 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:03.691606 29859 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.796798 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:03.803097 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:03.824469 24955 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:03.825173 24953 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 13 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20240426 13:21:03.797753 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:03.772321 29666 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:03.742182 29966 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:04.322352 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.106s	sys 0.170s
W20240426 13:21:04.352078 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.109s	sys 0.173s
I20240426 13:21:03.825543 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:04.219394 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2086) took 2015 ms (client timeout 2036 ms). Trace:
I20240426 13:21:04.319200 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 846) took 1410 ms. Trace:
I20240426 13:21:04.353528 25104 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
W20240426 13:21:04.353464 24950 rpcz_store.cc:269] 0426 13:21:02.203547 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.204274 (+   727us) service_pool.cc:225] Handling call
0426 13:21:04.219194 (+2014920us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.219320 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405124352}
I20240426 13:21:04.021698 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1905) took 2842 ms. Trace:
W20240426 13:21:03.828475 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54290.1us
I20240426 13:21:03.815979 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2703) took 2233 ms. Trace:
I20240426 13:21:04.308421 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2097) took 1464 ms. Trace:
I20240426 13:21:04.355156 25097 rpcz_store.cc:276] 0426 13:21:01.179353 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.179514 (+   161us) service_pool.cc:225] Handling call
0426 13:21:01.179776 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.166113 (+986337us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.021471 (+1855358us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:04.021631 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2246292352}
I20240426 13:21:04.358954 24952 rpcz_store.cc:276] 0426 13:21:02.844322 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.846515 (+  2193us) service_pool.cc:225] Handling call
0426 13:21:04.308241 (+1461726us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54002ca6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.308367 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":588582656}
I20240426 13:21:04.360669 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:04.361992 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:04.264400 25022 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:04.222054 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1924) took 1405 ms. Trace:
I20240426 13:21:03.824455 25023 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:04.367540 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1932) took 2215 ms. Trace:
I20240426 13:21:04.370160 25023 rpcz_store.cc:276] 0426 13:21:02.151966 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.819773 (+1667807us) service_pool.cc:225] Handling call
0426 13:21:03.820019 (+   246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.367248 (+547229us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.367481 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.355589 25029 rpcz_store.cc:276] 0426 13:21:01.582861 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.814706 (+2231845us) service_pool.cc:225] Handling call
0426 13:21:03.815043 (+   337us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.815724 (+   681us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.815922 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.394618 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:04.395288 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1933) took 1968 ms. Trace:
I20240426 13:21:04.395465 24955 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:04.395671 25022 rpcz_store.cc:276] 0426 13:21:02.426883 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.263712 (+1836829us) service_pool.cc:225] Handling call
0426 13:21:04.264153 (+   441us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.395032 (+130879us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.395239 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.365012 25102 rpcz_store.cc:276] 0426 13:21:02.816997 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.840513 (+ 23516us) service_pool.cc:225] Handling call
0426 13:21:02.840765 (+   252us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:02.840961 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:02.846045 (+  5084us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.374916 (+528871us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b1800558d00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:21:03.503972 (+129056us) log.cc:842] Serialized 14 byte log entry
0426 13:21:04.221342 (+717370us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b1800558d00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8ffa84e3 00007fda8ff9deaa 00007fda8ff9e3fb 00007fda90321a1a 00007fda9033109f 00007fda90331018 00007fda90330f98
0426 13:21:04.221615 (+   273us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.221700 (+    85us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:04.221768 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.221869 (+   101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.221993 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":641891,"prepare.queue_time_us":10173,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":4046,"raft.queue_time_us":1679,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":94,"spinlock_wait_cycles":8257664,"thread_start_us":126953,"threads_started":1,"wal-append.queue_time_us":656017}
I20240426 13:21:04.203449 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 930) took 1309 ms. Trace:
W20240426 13:21:04.440620 25070 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.025s	sys 0.054s
I20240426 13:21:03.772439 29775 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:04.353821 25619 rpcz_store.cc:276] 0426 13:21:02.908339 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.910301 (+  1962us) service_pool.cc:225] Handling call
0426 13:21:02.910585 (+   284us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.033077 (+122492us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:03.178418 (+145341us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.288391 (+109973us) log.cc:842] Serialized 14 byte log entry
0426 13:21:03.296173 (+  7782us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:03.296210 (+    37us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:03.296247 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.471332 (+175085us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b180047ba80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351
0426 13:21:03.472820 (+  1488us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b180047ba80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:04.049409 (+576589us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b180047ba80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:04.318927 (+269518us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54005c1054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:04.318983 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.319129 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":258525,"prepare.queue_time_us":236946,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":16702,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":15032,"spinlock_wait_cycles":346334208,"thread_start_us":216034,"threads_started":2,"wal-append.queue_time_us":167670}
W20240426 13:21:04.471297 25070 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.031s	sys 0.064s
I20240426 13:21:03.940610 25098 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:04.454977 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:04.401679 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:04.501547 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 729687us
I20240426 13:21:04.207367 25031 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:21:04.447242 24955 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:04.477294 25023 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:04.217710 29986 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.561s	user 0.000s	sys 0.001s
I20240426 13:21:04.503446 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 642) took 2192 ms. Trace:
I20240426 13:21:04.085695 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2094) took 1242 ms. Trace:
I20240426 13:21:04.127985 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 847) took 1219 ms. Trace:
I20240426 13:21:04.022415 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:04.440280 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.006s	sys 0.009s
I20240426 13:21:04.441721 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:04.214818 29990 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.433s	user 0.001s	sys 0.000s
I20240426 13:21:04.522120 25031 rpcz_store.cc:276] 0426 13:21:02.310781 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.206556 (+1895775us) service_pool.cc:225] Handling call
0426 13:21:04.206922 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.503209 (+296287us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.503397 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.523286 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.523655 25031 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20240426 13:21:04.524461 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 653) took 1268 ms. Trace:
I20240426 13:21:04.524796 25031 rpcz_store.cc:276] 0426 13:21:03.255879 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.523139 (+1267260us) service_pool.cc:225] Handling call
0426 13:21:04.523383 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.524256 (+   873us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.524415 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.525635 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:04.526216 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1955) took 1258 ms. Trace:
I20240426 13:21:04.400259 25616 rpcz_store.cc:276] 0426 13:21:02.893683 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.894140 (+   457us) service_pool.cc:225] Handling call
0426 13:21:02.894443 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.141662 (+247219us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.203167 (+1061505us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:04.203382 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":561107200}
I20240426 13:21:04.526645 25031 rpcz_store.cc:276] 0426 13:21:03.267851 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.525489 (+1257638us) service_pool.cc:225] Handling call
0426 13:21:04.525774 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.525931 (+   157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:04.525956 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.526002 (+    46us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:04.526037 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.526083 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.526172 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:03.961925 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:03.796502 29983 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
W20240426 13:21:03.799248 29987 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20240426 13:21:04.503033 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.293493 29179 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:04.207970 29960 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.618s	user 0.005s	sys 0.003s
I20240426 13:21:04.083683 29894 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:04.545905 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:04.510947 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1943) took 1639 ms. Trace:
I20240426 13:21:04.547276 25023 rpcz_store.cc:276] 0426 13:21:02.871562 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.372860 (+1501298us) service_pool.cc:225] Handling call
0426 13:21:04.469254 (+ 96394us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.508917 (+ 39663us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.510847 (+  1930us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.513075 24956 rpcz_store.cc:276] 0426 13:21:02.843522 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.843618 (+    96us) service_pool.cc:225] Handling call
0426 13:21:04.085505 (+1241887us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b540030c0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.085632 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91894528}
I20240426 13:21:04.513972 25620 rpcz_store.cc:276] 0426 13:21:02.908635 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.909050 (+   415us) service_pool.cc:225] Handling call
0426 13:21:02.909329 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.508342 (+599013us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.127664 (+619322us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:04.127909 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1321137920}
I20240426 13:21:03.822548 29058 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:04.056012 29961 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.829s	user 0.005s	sys 0.005s
I20240426 13:21:04.018226 29643 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:03.789732 29683 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:03.998888 29351 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:04.512053 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.007s	sys 0.010s
I20240426 13:21:04.477337 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:04.325340 29218 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.001s	sys 0.008s
I20240426 13:21:04.518658 25619 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20240426 13:21:04.476475 29954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.016s	sys 0.022s
I20240426 13:21:04.531507 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:04.532292 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 935) took 1538 ms (client timeout 2017 ms). Trace:
W20240426 13:21:04.616632 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85220.9us
W20240426 13:21:04.505326 29908 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.846s	user 0.008s	sys 0.001s
I20240426 13:21:04.528829 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2705) took 1168 ms. Trace:
I20240426 13:21:04.638855 25031 rpcz_store.cc:276] 0426 13:21:03.360738 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.527888 (+1167150us) service_pool.cc:225] Handling call
0426 13:21:04.528149 (+   261us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.528301 (+   152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:04.528325 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.528359 (+    34us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:04.528393 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.528439 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.528775 (+   336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.640700 25031 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:04.641311 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2715) took 1273 ms. Trace:
I20240426 13:21:04.641659 25031 rpcz_store.cc:276] 0426 13:21:03.367401 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.640102 (+1272701us) service_pool.cc:225] Handling call
0426 13:21:04.640477 (+   375us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.641071 (+   594us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.641258 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.533267 24952 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:04.024871 29876 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:04.084236 29993 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.313s	user 0.001s	sys 0.000s
I20240426 13:21:04.313993 29776 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:04.025879 29909 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:04.204950 29421 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:04.250900 28777 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:04.407441 29534 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:04.504000 29874 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:04.681106 29874 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.469956 29801 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:03.783116 29872 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.296109 29759 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:21:04.569296 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:04.554016 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.691691 24953 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20240426 13:21:04.571848 24950 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:04.516505 29688 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:04.616722 25615 rpcz_store.cc:269] 0426 13:21:02.993569 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.993703 (+   134us) service_pool.cc:225] Handling call
0426 13:21:04.022274 (+1028571us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:04.532238 (+509964us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14527360}
I20240426 13:21:03.807401 29944 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.683784 29909 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:04.553942 29956 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.820s	user 0.010s	sys 0.001s
I20240426 13:21:03.786116 29989 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.617535 29961 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:04.610716 29218 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.001s	sys 0.011s
I20240426 13:21:04.622951 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:04.704988 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82116.1us
I20240426 13:21:04.705854 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:04.707106 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:04.623648 29894 raft_consensus.cc:2741] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20240426 13:21:04.624174 29862 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.881s	user 0.007s	sys 0.002s
W20240426 13:21:04.624847 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 92846.1us
I20240426 13:21:03.803601 29790 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:04.616269 29954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.016s	sys 0.022s
I20240426 13:21:04.628221 29503 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.628984 29775 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:04.629122 29677 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:04.630102 24955 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:04.545969 29960 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.617920 29351 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:04.634647 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 944) took 1147 ms. Trace:
I20240426 13:21:04.643730 25023 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:03.877068 29937 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.733s
I20240426 13:21:04.316262 29745 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:04.656889 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2087) took 2453 ms (client timeout 2503 ms). Trace:
W20240426 13:21:04.755419 24948 rpcz_store.cc:269] 0426 13:21:02.203839 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.203947 (+   108us) service_pool.cc:225] Handling call
0426 13:21:04.656655 (+2452708us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54003c4bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.656817 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":758800256}
I20240426 13:21:04.659206 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1960) took 1060 ms. Trace:
I20240426 13:21:04.756539 25031 rpcz_store.cc:276] 0426 13:21:03.598398 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.658184 (+1059786us) service_pool.cc:225] Handling call
0426 13:21:04.658576 (+   392us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.658803 (+   227us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:04.658838 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.658892 (+    54us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:04.658936 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.659002 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.659136 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.758023 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:04.758766 25031 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:04.398944 29932 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.610951 29666 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.644344 29908 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:04.616042 29630 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20240426 13:21:04.676635 29896 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:03.922550 29678 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.685207 25096 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:04.686218 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 179067us
I20240426 13:21:04.789214 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.790055 24948 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:04.657948 29871 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.545819 29757 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
W20240426 13:21:04.570056 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 175272us
W20240426 13:21:04.779376 29560 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.006s	user 0.010s	sys 0.000s
I20240426 13:21:04.709236 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:04.803864 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94717.2us
I20240426 13:21:04.804575 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:04.805366 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:04.704809 29805 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.882s	user 0.013s	sys 0.025s
I20240426 13:21:04.687932 29534 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.717016 29862 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.799314 29871 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.815706 29351 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.718266 29961 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.816745 29871 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:04.817379 29871 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:04.816773 29862 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.822140 29909 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.823374 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:04.719514 24947 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:04.823855 29782 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:04.719861 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2691) took 3950 ms (client timeout 1736 ms). Trace:
W20240426 13:21:04.824535 25027 rpcz_store.cc:269] 0426 13:21:00.769627 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:00.845700 (+ 76073us) service_pool.cc:225] Handling call
0426 13:21:04.719648 (+3873948us) spinlock_profiling.cc:243] Waited 2.77 s on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.719789 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6370893568}
I20240426 13:21:04.824509 29782 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.826017 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:04.827104 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:04.702428 29688 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.828686 29944 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.829438 29688 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.829644 29862 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:04.829952 30007 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:04.720808 25029 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20240426 13:21:04.829432 30009 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:04.830700 29862 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:04.831038 30009 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.830700 25029 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:21:04.832366 25029 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20240426 13:21:04.832659 25029 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:04.832702 29989 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:04.832995 30009 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.696000 29758 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.833330 29989 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:04.833719 29989 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:04.833624 29758 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.694260 29874 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.729264 29872 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.732635 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:04.834553 29989 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:04.835438 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102932us
I20240426 13:21:04.735942 24953 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:04.740607 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2581) took 1797 ms. Trace:
I20240426 13:21:04.838228 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2588) took 1823 ms. Trace:
I20240426 13:21:04.838061 24955 rpcz_store.cc:276] 0426 13:21:02.942845 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.541495 (+1598650us) service_pool.cc:225] Handling call
0426 13:21:04.541778 (+   283us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.629896 (+ 88118us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b54001a48a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:21:04.739722 (+109826us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.739959 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8695168}
I20240426 13:21:04.838855 24953 rpcz_store.cc:276] 0426 13:21:03.014646 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.692299 (+1677653us) service_pool.cc:225] Handling call
0426 13:21:04.715574 (+ 23275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.837973 (+122399us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.838175 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.747565 25624 rpcz_store.cc:276] 0426 13:21:03.486941 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.490056 (+  3115us) service_pool.cc:225] Handling call
0426 13:21:04.634450 (+1144394us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.634582 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13673216}
I20240426 13:21:04.712460 29521 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:04.841122 29521 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.841569 29521 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.843545 29521 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:04.803983 29630 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20240426 13:21:04.845228 29775 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.846521 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:04.847676 24920 leader_election.cc:398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:21:04.847973 24920 leader_election.cc:401] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240426 13:21:04.848735 29960 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.848984 29630 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11.
I20240426 13:21:04.764101 29745 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.850116 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2099) took 1670 ms. Trace:
I20240426 13:21:04.850816 29745 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.851469 29876 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:04.852583 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:04.850474 24953 rpcz_store.cc:276] 0426 13:21:03.180158 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.839720 (+1659562us) service_pool.cc:225] Handling call
0426 13:21:04.840028 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.849830 (+  9802us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:04.849858 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.849896 (+    38us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:04.849928 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.849979 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.850069 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.855129 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.855957 24953 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20240426 13:21:04.857597 29894 raft_consensus.cc:2741] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20240426 13:21:04.857801 25027 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:04.859853 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:04.737174 29366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.020s	sys 0.050s
W20240426 13:21:04.860445 29366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.183s	user 0.020s	sys 0.050s
I20240426 13:21:04.766117 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2693) took 3972 ms. Trace:
I20240426 13:21:04.785311 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 634) took 4173 ms. Trace:
I20240426 13:21:04.861238 25030 rpcz_store.cc:276] 0426 13:21:00.612275 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.330205 (+2717930us) service_pool.cc:225] Handling call
0426 13:21:03.378899 (+ 48694us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:03.381796 (+  2897us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.785063 (+1403267us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b5400415ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:04.785243 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110925056}
I20240426 13:21:04.797295 25619 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:21:04.776166 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:04.801801 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:04.701584 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:04.863373 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61692.6us
I20240426 13:21:04.806525 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:04.810518 25025 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:21:04.863894 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 162392us
I20240426 13:21:04.863997 24948 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:04.864001 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:04.866168 29534 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.866261 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:04.761698 29966 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:04.799243 29386 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.016s
I20240426 13:21:04.745824 29503 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.812847 29446 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:04.718823 29421 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:04.837625 29757 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:04.798336 29896 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:04.801717 29908 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:04.860873 29985 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.213s	user 0.000s	sys 0.003s
I20240426 13:21:04.753624 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2706) took 1392 ms. Trace:
I20240426 13:21:04.862746 24954 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:04.708043 24951 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:04.720741 29683 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:04.861094 25026 rpcz_store.cc:276] 0426 13:21:00.795667 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.576459 (+2780792us) service_pool.cc:225] Handling call
0426 13:21:03.576775 (+   316us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:03.603275 (+ 26500us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:03.603305 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:03.614125 (+ 10820us) log.cc:842] Serialized 52 byte log entry
0426 13:21:03.742082 (+127957us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:03.742406 (+   324us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:03.742471 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:04.765857 (+1023386us) raft_consensus.cc:1671] finished
0426 13:21:04.765887 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.766051 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29974,"raft.queue_time_us":26144,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":8866,"thread_start_us":93816,"threads_started":1,"wal-append.queue_time_us":131325}
I20240426 13:21:04.804598 29948 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:04.869246 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:04.950292 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 81121us
I20240426 13:21:04.872609 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1962) took 1228 ms. Trace:
W20240426 13:21:04.811918 29805 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.014s	sys 0.025s
I20240426 13:21:04.877964 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:04.951258 25027 rpcz_store.cc:276] 0426 13:21:03.644301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.827873 (+1183572us) service_pool.cc:225] Handling call
0426 13:21:04.857644 (+ 29771us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.858203 (+   559us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.872561 (+ 14358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.951902 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.879853 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:04.877182 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1923) took 3792 ms. Trace:
I20240426 13:21:04.906169 25023 rpcz_store.cc:276] 0426 13:21:03.361030 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.549761 (+1188731us) service_pool.cc:225] Handling call
0426 13:21:04.643498 (+ 93737us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.753398 (+109900us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.753573 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.906442 25030 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20240426 13:21:04.930557 25098 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:21:04.932186 25097 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:21:04.933410 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:04.934893 29218 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:04.765511 29932 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:04.955307 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1963) took 1295 ms. Trace:
I20240426 13:21:04.876451 29961 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:04.976728 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:04.976908 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.085780 24953 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:04.905942 29801 raft_consensus.cc:2741] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:21:04.870568 29937 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:04.982101 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.105396 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:04.947356 29351 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.982146 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:04.983142 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:04.867848 29677 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:04.864125 29767 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:04.870301 29994 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:05.079913 25030 rpcz_store.cc:276] 0426 13:21:03.660066 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.862796 (+1202730us) service_pool.cc:225] Handling call
0426 13:21:04.906103 (+ 43307us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.955021 (+ 48918us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.955248 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.906863 29924 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.087247 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2593) took 1761 ms. Trace:
I20240426 13:21:04.952843 25028 rpcz_store.cc:276] 0426 13:21:01.085051 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.742948 (+2657897us) service_pool.cc:225] Handling call
0426 13:21:03.743264 (+   316us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:03.743575 (+   311us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:03.743605 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:03.756625 (+ 13020us) log.cc:842] Serialized 26 byte log entry
0426 13:21:04.610328 (+853703us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:04.870885 (+260557us) log.cc:842] Serialized 14 byte log entry
0426 13:21:04.876850 (+  5965us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:04.876918 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:04.876971 (+    53us) raft_consensus.cc:1671] finished
0426 13:21:04.876986 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:04.877091 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5727,"prepare.queue_time_us":160257,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":12493,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":8211,"thread_start_us":1027812,"threads_started":2,"wal-append.queue_time_us":864128}
I20240426 13:21:04.876428 29757 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:04.804697 29560 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.099988 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:05.102669 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:05.102959 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:04.951074 29421 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:04.982401 29938 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:04.887121 29534 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.209s
I20240426 13:21:05.081571 29909 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:04.983856 29643 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:05.113049 24953 rpcz_store.cc:276] 0426 13:21:03.325546 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.086474 (+1760928us) service_pool.cc:225] Handling call
0426 13:21:05.086799 (+   325us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.086944 (+   145us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.086968 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.087008 (+    40us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:05.087038 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.087085 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.087185 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:04.981628 29948 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:05.118263 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 312548us
W20240426 13:21:05.107656 30000 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20240426 13:21:05.105333 29446 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:05.144176 25028 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:21:04.943496 29908 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.145805 25103 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:05.173106 25103 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.125480 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.151680 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.169739 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 656) took 1435 ms. Trace:
I20240426 13:21:05.175271 25028 rpcz_store.cc:276] 0426 13:21:03.734171 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.119063 (+1384892us) service_pool.cc:225] Handling call
0426 13:21:05.119488 (+   425us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.169402 (+ 49914us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.169677 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.176596 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.112943 29961 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:05.174118 25103 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.159233 29446 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:05.110527 29351 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.110193 29677 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:05.162936 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.008s	sys 0.017s
I20240426 13:21:05.164367 29948 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.167598 29859 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.147868 29688 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:05.171875 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2102) took 1815 ms. Trace:
I20240426 13:21:05.117837 29757 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:05.178131 25103 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:05.178833 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:05.186851 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2093) took 2343 ms (client timeout 3011 ms). Trace:
I20240426 13:21:05.118899 29937 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:05.188951 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.009s	sys 0.017s
I20240426 13:21:05.190572 25026 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:05.207397 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.208556 24956 raft_consensus.cc:465] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Not starting forced leader election -- already a leader
I20240426 13:21:05.208808 29761 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:05.208906 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.209374 29761 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:05.209810 29761 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:05.210402 24996 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:05.211272 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.212474 24955 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:05.216399 25031 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20240426 13:21:05.217784 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2718) took 1442 ms. Trace:
I20240426 13:21:05.217958 25097 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:05.218195 25031 rpcz_store.cc:276] 0426 13:21:03.775668 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.182906 (+1407238us) service_pool.cc:225] Handling call
0426 13:21:05.203624 (+ 20718us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.217512 (+ 13888us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.217729 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.220438 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.185956 29759 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:05.201697 24949 rpcz_store.cc:269] 0426 13:21:02.843246 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.843346 (+   100us) service_pool.cc:225] Handling call
0426 13:21:05.186599 (+2343253us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:05.186777 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1289252608}
I20240426 13:21:05.200709 24953 rpcz_store.cc:276] 0426 13:21:03.356913 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.148272 (+1791359us) service_pool.cc:225] Handling call
0426 13:21:05.171318 (+ 23046us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.171503 (+   185us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.171537 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.171588 (+    51us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:05.171628 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.171698 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.171812 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.182296 29521 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:05.227631 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.223846 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:05.228588 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:05.192406 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 489461us
I20240426 13:21:05.155180 29576 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.115597 29801 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20240426 13:21:05.235114 25104 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:05.113214 29767 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.195232 29688 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:05.238047 25098 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:05.156270 29643 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.248775 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.250605 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.216552 29876 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:05.225710 29859 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:05.142534 29924 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.221493 29745 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:05.253057 24953 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 14. Ops: [12.7-12.7]
I20240426 13:21:05.225111 29421 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:05.259392 25031 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20240426 13:21:05.258901 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:05.252287 29688 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:05.264075 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2103) took 1891 ms. Trace:
I20240426 13:21:05.265411 25615 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:21:05.238426 29446 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.229597 30009 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20240426 13:21:05.268275 24953 rpcz_store.cc:276] 0426 13:21:03.372467 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.235539 (+1863072us) service_pool.cc:225] Handling call
0426 13:21:05.252840 (+ 17301us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.263813 (+ 10973us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.264008 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.275435 30009 consensus_queue.cc:1225] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240426 13:21:05.275740 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:05.264434 30001 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20240426 13:21:05.277616 25103 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:05.251606 29521 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:05.264495 29859 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:05.272964 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2105) took 1743 ms. Trace:
I20240426 13:21:05.254609 30019 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20240426 13:21:05.279220 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.263540 29503 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.328227 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:05.295984 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 214571us
I20240426 13:21:05.281658 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.331641 29643 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.279829 29960 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20240426 13:21:05.333038 29908 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.336370 30019 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:21:05.336542 30012 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:21:05.336789 30019 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.337630 30019 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.338798 30019 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:05.257266 29896 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.259608 29961 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.340183 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.341365 29677 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.247514 29576 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.342713 30029 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:05.343710 29576 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.323591 24949 rpcz_store.cc:276] 0426 13:21:03.529458 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.254635 (+1725177us) service_pool.cc:225] Handling call
0426 13:21:05.272418 (+ 17783us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.272625 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.272660 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.272716 (+    56us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:05.272762 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.272830 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.272918 (+    88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.345842 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.347357 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 848) took 1617 ms. Trace:
I20240426 13:21:05.258713 29876 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.337917 30028 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:05.347642 24949 rpcz_store.cc:276] 0426 13:21:03.730175 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.346629 (+1616454us) service_pool.cc:225] Handling call
0426 13:21:05.346909 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.347068 (+   159us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.347092 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.347134 (+    42us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:05.347172 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.347221 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.347308 (+    87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.349375 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 849) took 1619 ms. Trace:
I20240426 13:21:05.349407 29876 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.349923 24949 rpcz_store.cc:276] 0426 13:21:03.730360 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.348682 (+1618322us) service_pool.cc:225] Handling call
0426 13:21:05.348947 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.349092 (+   145us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.349118 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.349156 (+    38us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:05.349187 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.349235 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.349330 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.351388 24949 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
W20240426 13:21:05.340639 30024 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20240426 13:21:05.352545 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 857) took 1579 ms (client timeout 1844 ms). Trace:
W20240426 13:21:05.353010 24951 rpcz_store.cc:269] 0426 13:21:03.773511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.816688 (+ 43177us) service_pool.cc:225] Handling call
0426 13:21:05.352365 (+1535677us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:05.352483 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2047444224}
I20240426 13:21:05.354720 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 852) took 1614 ms. Trace:
I20240426 13:21:05.355120 24951 rpcz_store.cc:276] 0426 13:21:03.740563 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.353801 (+1613238us) service_pool.cc:225] Handling call
0426 13:21:05.354127 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.354323 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.354376 (+    53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.354428 (+    52us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:05.354481 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.354544 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.354663 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.356621 25098 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:05.351922 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 850) took 1621 ms. Trace:
I20240426 13:21:05.280447 30009 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:05.357892 24949 rpcz_store.cc:276] 0426 13:21:03.730517 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.350932 (+1620415us) service_pool.cc:225] Handling call
0426 13:21:05.351220 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.351707 (+   487us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.351876 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.269133 30017 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:05.279237 25026 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:05.301017 29446 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:05.365026 30023 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20240426 13:21:05.276307 29787 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20240426 13:21:05.323498 29890 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:05.325588 29932 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.375149 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20240426 13:21:05.380584 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 132107us
I20240426 13:21:05.267108 29745 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.381366 25104 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:21:05.382522 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 853) took 1639 ms. Trace:
I20240426 13:21:05.328068 29560 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.381011 24949 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 12. (term mismatch)
I20240426 13:21:05.384114 24951 rpcz_store.cc:276] 0426 13:21:03.742664 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.362215 (+1619551us) service_pool.cc:225] Handling call
0426 13:21:05.374769 (+ 12554us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.382305 (+  7536us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.382473 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.386610 24949 pending_rounds.cc:76] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 11
I20240426 13:21:05.385123 25105 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:05.387109 24949 pending_rounds.cc:97] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20240426 13:21:05.388828 24949 raft_consensus.cc:2866] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:05.389765 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 865) took 1601 ms. Trace:
I20240426 13:21:05.390198 24949 rpcz_store.cc:276] 0426 13:21:03.789234 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.363962 (+1574728us) service_pool.cc:225] Handling call
0426 13:21:05.380683 (+ 16721us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.389510 (+  8827us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.389717 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:05.391389 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.373956 29956 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.394945 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.396050 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:05.395188 25101 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:21:05.397157 25619 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Deduplicated request from leader. Original: 14.12->[16.13-16.13]   Dedup: 16.13->[]
I20240426 13:21:05.378825 30017 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.385391 29787 consensus_queue.cc:1225] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 9.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:21:05.279580 29937 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.378027 29685 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
W20240426 13:21:05.425212 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 478892us
W20240426 13:21:05.396322 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147608us
I20240426 13:21:05.404417 29560 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.427352 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:05.427767 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 216574us
I20240426 13:21:05.405259 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.405454 25615 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:21:05.405992 25619 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
W20240426 13:21:05.406639 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 179090us
I20240426 13:21:05.388937 29351 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.409560 25620 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:05.407994 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.387627 29688 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.408514 29677 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:05.436684 24921 connection.cc:232] RPC call timeout handler was delayed by -1.015981970000098s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20240426 13:21:05.438071 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.439738 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.432121 24951 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:05.397348 30028 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20240426 13:21:05.388398 30022 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20240426 13:21:05.392513 29896 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20240426 13:21:05.403932 29944 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.032s
I20240426 13:21:05.441864 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:05.447827 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.448446 25615 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:05.424260 29767 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.436321 29876 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:05.444355 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:05.423446 29782 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.395345 29994 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20240426 13:21:05.404243 29956 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.447316 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2107) took 1759 ms. Trace:
I20240426 13:21:05.435288 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:05.449623 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:05.427839 30009 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:05.451297 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:05.508625 24949 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:05.407125 29666 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20240426 13:21:05.452596 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.503646 24953 rpcz_store.cc:276] 0426 13:21:03.688173 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.276607 (+1588434us) service_pool.cc:225] Handling call
0426 13:21:05.317721 (+ 41114us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.317941 (+   220us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:05.327976 (+ 10035us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.446929 (+118953us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.446990 (+    61us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:05.447052 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.447128 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.447262 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10346,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":70,"raft.run_cpu_time_us":8476,"raft.run_wall_time_us":50831,"thread_start_us":118627,"threads_started":1}
I20240426 13:21:05.453119 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.437229 29560 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:05.512177 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 621) took 3377 ms (client timeout 2036 ms). Trace:
I20240426 13:21:05.510561 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.514091 24951 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:05.544862 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109623us
I20240426 13:21:05.545379 25105 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20240426 13:21:05.520715 25618 rpcz_store.cc:269] 0426 13:21:02.134245 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.134366 (+   121us) service_pool.cc:225] Handling call
0426 13:21:05.511935 (+3377569us) spinlock_profiling.cc:243] Waited 2.69 s on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:05.512075 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6177069952}
I20240426 13:21:05.442740 29677 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.442438 29429 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20240426 13:21:05.549676 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.555873 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:05.557564 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52115.6us
I20240426 13:21:05.451197 29685 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20240426 13:21:05.452225 29938 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.062s
I20240426 13:21:05.444326 30018 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20240426 13:21:05.508698 30036 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:05.511373 30035 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:05.544873 29876 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.586783 29876 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.587769 29961 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:05.588587 29876 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.568078 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.589928 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.591202 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.557490 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:05.566226 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:05.565236 30016 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20240426 13:21:05.506995 29761 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.606165 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:05.596287 30035 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.514207 30009 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.617640 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.568691 30031 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:05.618377 30047 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:05.618495 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.618768 24949 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:05.597559 29876 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:05.596978 30028 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:05.601014 29896 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
W20240426 13:21:05.620460 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74568us
I20240426 13:21:05.624259 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.627369 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:05.571079 29787 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20240426 13:21:05.511404 30017 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.636266 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.637769 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2702) took 4061 ms. Trace:
W20240426 13:21:05.637928 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 51010.3us
I20240426 13:21:05.638971 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.638149 25024 rpcz_store.cc:276] 0426 13:21:01.576038 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.782685 (+2206647us) service_pool.cc:225] Handling call
0426 13:21:03.827789 (+ 45104us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:04.400596 (+572807us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:21:04.684900 (+284304us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.118590 (+433690us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:05.150656 (+ 32066us) log.cc:842] Serialized 26 byte log entry
0426 13:21:05.150833 (+   177us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:05.151062 (+   229us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.151230 (+   168us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.151303 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:05.637574 (+486271us) raft_consensus.cc:1671] finished
0426 13:21:05.637594 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.637718 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47474,"prepare.queue_time_us":336562,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5970,"raft.queue_time_us":526911,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":65476,"spinlock_wait_cycles":12283776,"thread_start_us":623677,"threads_started":2,"wal-append.queue_time_us":582285}
I20240426 13:21:05.584511 30006 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20240426 13:21:05.640729 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.644302 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.648479 24953 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:05.624054 30031 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:05.650691 24949 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:21:05.655501 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.656325 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:05.635033 30018 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.666188 24951 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:05.666703 24953 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:21:05.666093 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:05.667304 24953 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:05.645386 29790 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:05.666659 24951 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:05.668479 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.667348 29688 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.674180 29775 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:05.674640 29775 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:05.674974 29775 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:05.676802 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.677961 30046 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.678537 30046 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.680166 29503 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:05.680934 30046 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.681581 30046 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:05.682026 30046 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.682991 29989 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:05.684679 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2073) took 4423 ms (client timeout 1590 ms). Trace:
W20240426 13:21:05.685065 24954 rpcz_store.cc:269] 0426 13:21:01.248511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:01.248689 (+   178us) service_pool.cc:225] Handling call
0426 13:21:05.670866 (+4422177us) spinlock_profiling.cc:243] Waited 3.6 s on lock 0x7b54001d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:05.671961 (+  1095us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8278821760}
I20240426 13:21:05.686496 24954 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:05.686805 24954 pending_rounds.cc:76] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 7
I20240426 13:21:05.686993 24954 pending_rounds.cc:97] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:05.687201 24954 raft_consensus.cc:2866] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:05.665920 30011 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20240426 13:21:05.690946 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.691879 30009 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.692004 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.692706 29872 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:05.693485 29872 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:21:05.694049 29872 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:05.697520 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 247998us
I20240426 13:21:05.697739 24954 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:05.698462 24954 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:05.701509 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:05.698390 24953 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.704479 24953 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:05.682963 30046 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.651790 29896 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:05.699362 24954 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.705140 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.657509 29966 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:21:05.663897 30030 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20240426 13:21:05.673243 24951 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.657785 30045 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.657750 30021 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.631259 29876 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.672787 29790 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:05.708302 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:05.708753 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.623559 30044 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20240426 13:21:05.713941 24954 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:05.715052 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:05.715927 24951 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:05.629360 30047 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.658448 30031 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:05.711414 29956 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.637409 30028 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.729735 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.741194 24953 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240426 13:21:05.741271 24954 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20240426 13:21:05.730674 24852 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:05.742156 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:04.086555 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:05.732849 29677 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:05.756374 30039 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20240426 13:21:05.757230 30038 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20240426 13:21:05.758476 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2100) took 2577 ms. Trace:
I20240426 13:21:05.722797 30045 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.758877 24948 rpcz_store.cc:276] 0426 13:21:03.180696 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.981810 (+1801114us) service_pool.cc:225] Handling call
0426 13:21:05.103920 (+122110us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.110472 (+  6552us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.110502 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:05.144008 (+ 33506us) log.cc:842] Serialized 26 byte log entry
0426 13:21:05.344487 (+200479us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:21:05.441331 (+ 96844us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:05.441385 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.441467 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:05.758187 (+316720us) raft_consensus.cc:1671] finished
0426 13:21:05.758212 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.758402 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6112,"raft.queue_time_us":6677,"raft.run_cpu_time_us":2001,"raft.run_wall_time_us":231652,"spinlock_wait_cycles":64307968,"thread_start_us":272050,"threads_started":1,"wal-append.queue_time_us":365908}
W20240426 13:21:05.760452 30040 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20240426 13:21:05.760803 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:05.761281 30045 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.722698 30021 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.761407 24948 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:05.763617 30047 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.764204 30028 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.743273 30044 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:05.733433 29688 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:21:05.717550 30032 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20240426 13:21:05.733999 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:05.704501 29757 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.707331 29767 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.734031 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:05.716253 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.710016 29761 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.714418 29759 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.671121 29745 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.671345 30018 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:05.790490 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74332.3us
I20240426 13:21:05.728871 29790 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:05.756289 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 652) took 2543 ms. Trace:
I20240426 13:21:05.732893 29876 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.772943 24953 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:05.774397 29775 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.799712 25027 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:05.741123 29560 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.775470 30021 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.779145 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2574) took 2861 ms. Trace:
I20240426 13:21:05.790664 29557 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:05.804502 24947 rpcz_store.cc:276] 0426 13:21:02.917195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.923957 (+  6762us) service_pool.cc:225] Handling call
0426 13:21:04.085633 (+1161676us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.824283 (+738650us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.195818 (+371535us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b540030c0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:05.778907 (+583089us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540030c0d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:05.779069 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4073020928}
I20240426 13:21:05.797155 29759 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.808655 29759 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.809065 29757 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.810827 29775 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:05.812781 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:05.741840 24954 pending_rounds.cc:76] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:21:05.814738 24954 pending_rounds.cc:97] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20240426 13:21:05.815054 24954 raft_consensus.cc:2866] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:05.816731 24953 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:05.799633 29688 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.791126 29767 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.817981 29688 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:05.819242 29767 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:05.783596 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 120573us
I20240426 13:21:05.781337 29896 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.787144 25024 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:05.720410 30041 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:05.794533 25022 rpcz_store.cc:276] 0426 13:21:03.212975 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.439555 (+1226580us) service_pool.cc:225] Handling call
0426 13:21:04.507310 (+ 67755us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:04.522112 (+ 14802us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:04.877581 (+355469us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.301896 (+424315us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:05.321250 (+ 19354us) log.cc:842] Serialized 52 byte log entry
0426 13:21:05.321410 (+   160us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:05.321655 (+   245us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.322047 (+   392us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.322409 (+   362us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.322546 (+   137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:05.691575 (+369029us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b18002c2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:05.756104 (+ 64529us) raft_consensus.cc:1671] finished
0426 13:21:05.756125 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.756239 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":248109,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":80,"raft.queue_time_us":48269,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":6092,"spinlock_wait_cycles":89377024,"thread_start_us":494354,"threads_started":2,"wal-append.queue_time_us":304236}
I20240426 13:21:05.798840 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.822962 25100 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:05.824399 24947 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:05.825290 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:05.817626 30018 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:05.788877 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.829972 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:05.831350 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.831615 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.788244 30028 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20240426 13:21:05.832234 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:05.819891 29688 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:05.829509 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:05.834491 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:05.826624 29954 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
W20240426 13:21:05.827610 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 135661us
I20240426 13:21:05.829046 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.838201 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.840621 25624 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:05.842408 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.824314 30041 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:05.847548 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:05.917567 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:05.918596 24954 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:05.927264 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79752.4us
I20240426 13:21:05.932441 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:05.934096 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95065.1us
I20240426 13:21:05.938161 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:05.939795 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.940464 25024 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:05.941259 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:05.831249 29759 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:05.943445 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.948762 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:05.949308 25022 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:05.949616 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.950892 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.951104 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:05.952893 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.953541 24948 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:05.954391 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:05.955204 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.827366 30009 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.107s
I20240426 13:21:05.963040 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:05.963824 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2717) took 2188 ms. Trace:
I20240426 13:21:05.959903 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.958655 29446 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.002s
I20240426 13:21:05.950677 29757 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:05.973994 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:05.967231 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.831007 29645 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.117s
I20240426 13:21:05.915004 30041 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:05.964306 25028 rpcz_store.cc:276] 0426 13:21:03.775519 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.175938 (+1400419us) service_pool.cc:225] Handling call
0426 13:21:05.176234 (+   296us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.176518 (+   284us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.176545 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:05.188135 (+ 11590us) log.cc:842] Serialized 26 byte log entry
0426 13:21:05.379450 (+191315us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:05.381467 (+  2017us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.381803 (+   336us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.381893 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:05.963625 (+581732us) raft_consensus.cc:1671] finished
0426 13:21:05.963644 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:05.963759 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":110,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":3699,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":11917,"spinlock_wait_cycles":4493056,"thread_start_us":178691,"threads_started":1,"wal-append.queue_time_us":249982}
I20240426 13:21:05.994760 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:05.997917 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.998962 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.943178 30021 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
W20240426 13:21:05.989615 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48405.6us
I20240426 13:21:05.917879 29989 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
W20240426 13:21:06.037317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50075.8us
I20240426 13:21:05.968484 29761 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:05.933593 29560 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20240426 13:21:05.979027 29908 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:06.038322 25022 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:21:06.041774 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87441.9us
I20240426 13:21:05.958487 29759 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.064007 29446 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.065877 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.049571 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:06.002996 29975 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.005s	user 0.022s	sys 0.056s
I20240426 13:21:05.999460 29872 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20240426 13:21:06.039443 29876 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20240426 13:21:06.052381 30056 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:05.914774 30057 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.196s
I20240426 13:21:06.054345 29568 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:06.054885 29937 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20240426 13:21:06.126597 30057 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:21:06.055915 29924 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:06.058874 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:06.127498 29521 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.127983 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:06.066742 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:06.123864 25618 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:05.979743 29767 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:21:06.072221 25022 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:06.132560 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 158632us
I20240426 13:21:06.133478 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.135434 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.131104 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:06.076108 25024 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:06.001139 29677 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:06.080868 29775 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20240426 13:21:06.077512 29975 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.079s	user 0.023s	sys 0.057s
I20240426 13:21:06.090369 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.094357 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1945) took 3197 ms. Trace:
W20240426 13:21:06.145212 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54896.7us
I20240426 13:21:06.145069 25029 rpcz_store.cc:276] 0426 13:21:02.897301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.373250 (+1475949us) service_pool.cc:225] Handling call
0426 13:21:04.720472 (+347222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.833237 (+112765us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.094181 (+1260944us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54003f6e50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:06.094301 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797700480}
I20240426 13:21:06.146798 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.147066 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.098215 29989 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:06.098209 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2719) took 2322 ms. Trace:
I20240426 13:21:06.150768 25031 rpcz_store.cc:276] 0426 13:21:03.775823 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.219218 (+1443395us) service_pool.cc:225] Handling call
0426 13:21:05.237848 (+ 18630us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:05.263753 (+ 25905us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.097953 (+834200us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.098142 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48737536}
I20240426 13:21:06.064642 29446 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.103052 29758 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.078861 30031 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:05.994269 29757 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:06.059590 29790 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.064841 29759 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:06.121596 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:05.967979 29956 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
W20240426 13:21:06.138254 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139494us
I20240426 13:21:06.148267 30057 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:21:06.139591 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 176639us
I20240426 13:21:06.169180 25028 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:06.169577 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.148761 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.170456 25029 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:06.151238 29568 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:06.061689 29761 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.157536 25105 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:06.157779 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.136456 30056 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.161551 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.139065 29677 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.139537 30009 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:06.166649 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.128013 29521 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.161201 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:06.144605 29682 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:06.207099 30075 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:06.207947 30075 consensus_queue.cc:1225] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240426 13:21:06.208148 25031 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:06.198819 29876 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:06.180009 29761 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:06.209921 29961 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:06.210244 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.211310 30043 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.144660 29924 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:06.211838 30043 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.139281 29775 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:06.168713 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:06.171150 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.212023 29924 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.217084 29924 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:06.212217 30041 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.219427 30041 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.213011 30050 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:06.220429 30050 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:06.221246 30050 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.211338 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.214602 30033 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:06.173487 24948 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:21:06.174953 24953 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:06.150478 29989 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:06.164736 29758 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:06.198038 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.201248 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.203635 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:06.150286 29937 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20240426 13:21:06.179875 30056 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:06.206970 29521 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.196578 29677 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:06.062382 30021 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:06.169579 29872 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:06.248885 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:06.216230 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:06.270177 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:06.216472 25029 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:06.222347 30050 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:06.235239 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 831) took 3439 ms. Trace:
W20240426 13:21:06.250756 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47228.5us
I20240426 13:21:06.276250 25103 rpcz_store.cc:276] 0426 13:21:02.795949 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:02.835144 (+ 39195us) service_pool.cc:225] Handling call
0426 13:21:04.330671 (+1495527us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:21:04.330930 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.319114 (+988184us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.234987 (+915873us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.235170 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1884965376}
I20240426 13:21:06.277470 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.252060 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:06.277942 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:06.278239 24948 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:06.249815 25022 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:06.291698 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:06.294152 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96190.4us
I20240426 13:21:06.295493 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:06.241123 30020 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.882s	user 0.004s	sys 0.000s
I20240426 13:21:06.213873 30043 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.321913 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 844) took 2730 ms. Trace:
I20240426 13:21:06.322335 25096 rpcz_store.cc:276] 0426 13:21:03.591459 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.592383 (+   924us) service_pool.cc:225] Handling call
0426 13:21:03.592743 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.794322 (+1201579us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.321688 (+1527366us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.321857 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2511789824}
I20240426 13:21:06.324383 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2591) took 3261 ms. Trace:
I20240426 13:21:06.286613 30033 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.324800 24955 rpcz_store.cc:276] 0426 13:21:03.062736 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.839656 (+1776920us) service_pool.cc:225] Handling call
0426 13:21:05.187548 (+347892us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.219857 (+ 32309us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.323498 (+1103641us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.324183 (+   685us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b540038ead0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:06.324326 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":855734016}
I20240426 13:21:06.326103 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:06.327165 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:06.332100 25029 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
W20240426 13:21:06.298368 30062 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20240426 13:21:06.307518 29924 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.285393 29758 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.298647 30021 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.226846 28777 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:06.245498 29909 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:06.268882 29871 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:06.228675 30031 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:06.239243 29757 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:06.300113 30075 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:06.300282 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.304692 24948 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:06.355089 25031 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:06.305238 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.305672 30044 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:06.230361 29790 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.196463 29759 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:06.184118 30072 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.124s
I20240426 13:21:06.294447 30041 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20240426 13:21:06.209519 29446 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.284549 29773 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:06.309521 29989 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.310174 30056 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.244781 30029 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
W20240426 13:21:06.311396 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1973) took 1365 ms (client timeout 1602 ms). Trace:
W20240426 13:21:06.383195 25027 rpcz_store.cc:269] 0426 13:21:04.946412 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.981969 (+ 35557us) service_pool.cc:225] Handling call
0426 13:21:06.311157 (+1329188us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:06.311324 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1593025920}
I20240426 13:21:06.312453 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2589) took 3296 ms. Trace:
I20240426 13:21:06.313082 29179 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:06.313357 29557 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:06.315143 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.315712 30005 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:06.316085 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1949) took 1096 ms. Trace:
I20240426 13:21:06.386490 30033 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.386646 25100 rpcz_store.cc:276] 0426 13:21:05.219623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.219764 (+   141us) service_pool.cc:225] Handling call
0426 13:21:05.220065 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.823295 (+603230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.315850 (+492555us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.316013 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1385840512}
I20240426 13:21:06.316819 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2684) took 6208 ms. Trace:
I20240426 13:21:06.387630 25025 rpcz_store.cc:276] 0426 13:21:00.108573 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:03.098079 (+2989506us) service_pool.cc:225] Handling call
0426 13:21:03.124886 (+ 26807us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.863955 (+1739069us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.316604 (+1452649us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.316764 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3875582720}
I20240426 13:21:06.215274 29775 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:06.379946 30044 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:06.332355 24955 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:06.393335 30044 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.393991 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:06.395030 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:06.395905 30035 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.397177 30035 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:06.295032 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133535us
I20240426 13:21:06.384958 24950 rpcz_store.cc:276] 0426 13:21:03.015873 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.692934 (+1677061us) service_pool.cc:225] Handling call
0426 13:21:04.693237 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.765717 (+ 72480us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:04.765759 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:04.765800 (+    41us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:04.765836 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.312235 (+1546399us) spinlock_profiling.cc:243] Waited 72.3 ms on lock 0x7b5400102054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:06.312283 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:06.312399 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":166217472}
I20240426 13:21:06.401914 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:06.333779 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.403190 25029 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:06.349882 25620 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:06.350356 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:06.357059 25022 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:06.342108 29938 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20240426 13:21:06.364394 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:06.360740 29759 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.347201 28777 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:06.369271 25619 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
W20240426 13:21:06.372632 24996 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.911s (SENT)
I20240426 13:21:06.347380 29909 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:06.376091 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.415071 25031 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:06.380784 29773 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:06.381485 29989 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:06.381974 30056 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:06.398061 24955 pending_rounds.cc:76] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:21:06.417026 24955 pending_rounds.cc:97] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20240426 13:21:06.417363 24955 raft_consensus.cc:2866] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:06.406805 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.409767 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:06.412868 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 97765.5us
I20240426 13:21:06.413398 25619 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:06.413520 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:06.414031 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:06.422487 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.355333 29871 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.428606 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:06.422755 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50187us
I20240426 13:21:06.421736 25027 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:06.422719 25619 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.432535 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:06.432822 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.433703 25027 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:06.412477 29759 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:06.438221 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.438779 25027 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:06.414763 29790 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:06.415387 29682 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:06.479365 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.378129 29761 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.490387 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20240426 13:21:06.549538 24996 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 2.036s (SENT)
I20240426 13:21:06.550120 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:06.432163 29937 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:06.560873 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:06.432384 29859 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:21:06.582134 25027 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:06.576900 28777 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.433176 29890 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:06.435186 29446 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.613893 28777 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:06.614188 29446 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.616478 29446 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:06.617013 29934 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:21:06.619261 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:06.477555 29790 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:06.625052 29790 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:06.432093 30033 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:06.634152 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 211727us
W20240426 13:21:06.566814 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.565s (SENT) [suppressed 14 similar messages]
I20240426 13:21:06.634822 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.585398 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2587) took 3572 ms. Trace:
I20240426 13:21:06.600605 29630 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.595786 25031 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:06.414659 30011 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20240426 13:21:06.636803 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.637758 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 663) took 1205 ms. Trace:
I20240426 13:21:06.639079 25031 rpcz_store.cc:276] 0426 13:21:05.433655 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.636167 (+1202512us) service_pool.cc:225] Handling call
0426 13:21:06.637681 (+  1514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:06.657860 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:06.658788 25105 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:06.660214 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.661662 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.734805 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:06.737927 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.654120 25027 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:06.741852 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.748909 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.765038 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:06.765802 25031 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:06.768898 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.769518 25103 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:06.637050 29757 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:21:06.770474 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135711us
I20240426 13:21:06.657080 29366 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:06.775279 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:06.742915 25027 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:06.733808 29852 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.786614 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:06.797204 25031 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20240426 13:21:06.836470 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2720) took 3060 ms. Trace:
I20240426 13:21:06.837100 25031 rpcz_store.cc:276] 0426 13:21:03.775961 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.766517 (+2990556us) service_pool.cc:225] Handling call
0426 13:21:06.790661 (+ 24144us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:06.836109 (+ 45448us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.836399 (+   290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:06.628777 29677 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:06.844892 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.848337 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.767362 29924 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:06.848942 30054 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:21:06.849565 28777 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240426 13:21:06.849854 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74647us
I20240426 13:21:06.834183 30071 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:06.850332 30057 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:06.856215 25096 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:06.857494 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:06.733789 29630 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.773434 29566 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
I20240426 13:21:06.786633 29366 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:06.901047 28777 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.903009 25031 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:21:06.791090 29758 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:06.937551 30096 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20240426 13:21:06.937899 29874 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20240426 13:21:06.853127 30033 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:06.910915 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.941128 25096 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:06.793144 29852 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.904453 30071 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:06.637537 29909 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.920517 28777 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:06.922281 29805 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20240426 13:21:06.922467 29859 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:06.927619 30035 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.530s	user 0.007s	sys 0.000s
I20240426 13:21:06.929260 29677 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:06.760505 29773 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.929468 29446 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.074606 29446 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.036527 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:06.969698 29761 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.077772 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:06.916254 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58861.8us
I20240426 13:21:07.081377 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.081844 29852 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:07.034646 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 467628us
I20240426 13:21:07.038357 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.084084 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 876) took 1386 ms. Trace:
I20240426 13:21:06.920886 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 658) took 3135 ms. Trace:
I20240426 13:21:07.043381 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1964) took 3360 ms. Trace:
I20240426 13:21:07.084542 24947 rpcz_store.cc:276] 0426 13:21:05.697196 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.805445 (+108249us) service_pool.cc:225] Handling call
0426 13:21:05.805753 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.824848 (+ 19095us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.083562 (+1258714us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b54003679d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.083871 (+   309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42545408}
I20240426 13:21:07.036689 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.086099 25030 rpcz_store.cc:276] 0426 13:21:03.682352 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.118635 (+1436283us) service_pool.cc:225] Handling call
0426 13:21:05.160421 (+ 41786us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:05.180678 (+ 20257us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.180723 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:05.187562 (+  6839us) log.cc:842] Serialized 52 byte log entry
0426 13:21:05.449170 (+261608us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:05.504289 (+ 55119us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.507579 (+  3290us) log.cc:842] Serialized 14 byte log entry
0426 13:21:05.603051 (+ 95472us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:05.603118 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:06.986288 (+1383170us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b1800339e20. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:07.042702 (+ 56414us) raft_consensus.cc:1671] finished
0426 13:21:07.042755 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.043081 (+   326us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33336,"prepare.queue_time_us":97451,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":6847,"raft.queue_time_us":71425,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":50495,"spinlock_wait_cycles":7970560,"thread_start_us":320002,"threads_started":2,"wal-append.queue_time_us":403983}
I20240426 13:21:07.090029 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.090153 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 2585) took 4095 ms. Trace:
I20240426 13:21:07.050350 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:07.046623 25098 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.042s	sys 0.037s
I20240426 13:21:07.092159 25096 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:07.092304 25098 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.042s	sys 0.037s
I20240426 13:21:07.093904 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.939023 29366 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.095535 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:06.965893 29871 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:07.110596 25028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 664) took 1672 ms (client timeout 1931 ms). Trace:
I20240426 13:21:07.111375 25027 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:07.112241 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1965) took 2240 ms. Trace:
I20240426 13:21:07.112610 25027 rpcz_store.cc:276] 0426 13:21:04.871403 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.103966 (+2232563us) service_pool.cc:225] Handling call
0426 13:21:07.110766 (+  6800us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.111866 (+  1100us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.112158 (+   292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:06.970152 29909 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:06.944255 29566 consensus_queue.cc:1225] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 16.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20240426 13:21:06.981389 29989 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.554s	user 0.008s	sys 0.000s
I20240426 13:21:06.945111 30033 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:06.906744 30057 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:07.120510 30057 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:06.969053 30071 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:07.124198 25023 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:07.124245 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 858) took 1774 ms. Trace:
W20240426 13:21:07.111102 25028 rpcz_store.cc:269] 0426 13:21:05.436823 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.994496 (+557673us) service_pool.cc:225] Handling call
0426 13:21:07.104556 (+1110060us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54000564d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.109138 (+  4582us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":399193344}
I20240426 13:21:07.125964 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.126605 25028 raft_consensus.cc:2362] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12. Re-sending same reply.
I20240426 13:21:06.929901 30082 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.124858 25098 rpcz_store.cc:276] 0426 13:21:05.349735 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.368965 (+ 19230us) service_pool.cc:225] Handling call
0426 13:21:05.369282 (+   317us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:05.370287 (+  1005us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:06.130922 (+760635us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b18003469c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:21:06.348513 (+217591us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.093036 (+744523us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:07.094081 (+  1045us) log.cc:842] Serialized 52 byte log entry
0426 13:21:07.094392 (+   311us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:07.094444 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.094582 (+   138us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.116047 (+ 21465us) raft_consensus.cc:1671] finished
0426 13:21:07.123918 (+  7871us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.124140 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":632343,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":6565,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":4991,"spinlock_wait_cycles":3901312,"thread_start_us":935626,"threads_started":2,"wal-append.queue_time_us":628623}
I20240426 13:21:06.984751 29960 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:07.127861 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.129220 29960 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:07.130702 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.090747 24956 rpcz_store.cc:276] 0426 13:21:02.994108 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.569149 (+1575041us) service_pool.cc:225] Handling call
0426 13:21:07.089678 (+2520529us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9030ef37 00007fda9525bb50 00007fda8e949754 00007fda8e9496e2 00007fda8e949611 00007fda8e94958c 00007fda8e948842 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.089875 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1191780736}
W20240426 13:21:06.939242 25618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.016s	sys 0.050s
W20240426 13:21:07.136651 25618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.017s	sys 0.050s
I20240426 13:21:07.136449 29954 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.137512 24956 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:07.096385 29179 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.086124 25031 rpcz_store.cc:276] 0426 13:21:03.785809 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.838450 (+3052641us) service_pool.cc:225] Handling call
0426 13:21:06.902671 (+ 64221us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:06.920547 (+ 17876us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.920815 (+   268us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:07.099109 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2733) took 1457 ms (client timeout 1706 ms). Trace:
I20240426 13:21:07.100538 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2122) took 1323 ms. Trace:
W20240426 13:21:07.100880 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1972) took 2154 ms (client timeout 1565 ms). Trace:
W20240426 13:21:06.960620 29775 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.572s	user 0.008s	sys 0.001s
W20240426 13:21:07.140367 25026 rpcz_store.cc:269] 0426 13:21:04.946062 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.982867 (+ 36805us) service_pool.cc:225] Handling call
0426 13:21:07.100422 (+2117555us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.100642 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191912320}
I20240426 13:21:07.154491 29775 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.156463 24951 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:07.156531 29678 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:07.139108 24953 rpcz_store.cc:276] 0426 13:21:05.765591 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.817463 (+ 51872us) service_pool.cc:225] Handling call
0426 13:21:05.817754 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:06.249471 (+431717us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.088529 (+839058us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54000be854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.088954 (+   425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821070720}
I20240426 13:21:07.160948 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.161691 29775 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.143537 30082 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.166463 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:07.171185 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.091759 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.114502 25027 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:07.103847 24947 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:21:07.118615 29630 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.119552 28777 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.127816 29871 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:07.139019 25024 rpcz_store.cc:269] 0426 13:21:05.641366 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.967007 (+325641us) service_pool.cc:225] Handling call
0426 13:21:07.098604 (+1131597us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.098876 (+   272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89965440}
I20240426 13:21:07.130751 29960 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.132690 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:07.133756 30033 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.073892 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 10 to 13, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 13 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:07.141597 29179 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.142802 25031 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:07.143317 24956 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:07.152806 29989 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:07.153462 29761 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:07.159503 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:07.118292 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2337) took 1912 ms (client timeout 2036 ms). Trace:
I20240426 13:21:07.209883 24956 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.197438 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1966) took 2316 ms. Trace:
I20240426 13:21:07.212178 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1967) took 2339 ms. Trace:
I20240426 13:21:07.214483 25027 rpcz_store.cc:276] 0426 13:21:04.872555 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.113518 (+2240963us) service_pool.cc:225] Handling call
0426 13:21:07.114019 (+   501us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.185125 (+ 71106us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.188476 (+  3351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.214854 25031 rpcz_store.cc:276] 0426 13:21:04.872903 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.141855 (+2268952us) service_pool.cc:225] Handling call
0426 13:21:07.142308 (+   453us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.209773 (+ 67465us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.212106 (+  2333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.217180 29989 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.217399 29871 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.218349 25028 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20240426 13:21:07.213959 25101 rpcz_store.cc:269] 0426 13:21:05.205844 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.206017 (+   173us) service_pool.cc:225] Handling call
0426 13:21:07.117907 (+1911890us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.118102 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381383552}
I20240426 13:21:07.220546 29894 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:07.221349 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:07.222275 25101 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:07.223968 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:07.224754 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 143838us
I20240426 13:21:07.225733 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.227260 29960 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:07.227483 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.232968 25098 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.219645 29871 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.219985 25028 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20240426 13:21:07.242833 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1977) took 2152 ms. Trace:
I20240426 13:21:07.243305 25028 rpcz_store.cc:276] 0426 13:21:05.086332 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.219182 (+2132850us) service_pool.cc:225] Handling call
0426 13:21:07.219601 (+   419us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.238337 (+ 18736us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.238606 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.244305 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:07.219650 29989 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.249684 28777 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:07.245318 29894 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:07.239215 30082 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.254714 25030 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:07.260303 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.263466 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 657) took 3489 ms. Trace:
I20240426 13:21:07.274535 29630 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.254951 25104 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:07.266561 29446 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.256196 29179 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.257014 29871 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:07.264149 30021 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.865s	user 0.016s	sys 0.024s
I20240426 13:21:07.276896 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.278302 30057 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.279731 30071 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:07.280617 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53270.8us
I20240426 13:21:07.262105 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1946) took 2053 ms. Trace:
I20240426 13:21:07.255285 25027 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:07.281288 28777 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.268268 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:07.286005 29989 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.274737 24956 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.285027 25023 rpcz_store.cc:276] 0426 13:21:03.773749 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.140393 (+1366644us) service_pool.cc:225] Handling call
0426 13:21:07.107171 (+1966778us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.124835 (+ 17664us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.263246 (+138411us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54002ad350. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.263398 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4441060736}
I20240426 13:21:07.287885 24955 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:07.292402 29759 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.800s	user 0.001s	sys 0.020s
I20240426 13:21:07.261088 30047 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.292412 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 12, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 12 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:07.307248 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1968) took 2428 ms. Trace:
I20240426 13:21:07.309175 25024 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:07.309592 24956 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:07.309965 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1969) took 2436 ms. Trace:
I20240426 13:21:07.306970 30047 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.310449 25024 rpcz_store.cc:276] 0426 13:21:04.873575 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.237126 (+2363551us) service_pool.cc:225] Handling call
0426 13:21:07.298640 (+ 61514us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.309651 (+ 11011us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.309907 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.312728 30047 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:07.316183 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 145422us
I20240426 13:21:07.317042 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.317199 28777 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.308432 29630 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.320531 25105 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:07.320668 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.321776 29446 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:07.322093 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.322728 25023 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:07.307682 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1970) took 2422 ms. Trace:
I20240426 13:21:07.302973 25030 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:07.304733 25031 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20240426 13:21:07.304651 29876 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20240426 13:21:07.299494 30021 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.019s	sys 0.024s
I20240426 13:21:07.306057 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.305868 29989 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.326815 25024 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20240426 13:21:07.326418 25027 rpcz_store.cc:276] 0426 13:21:04.885456 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.215746 (+2330290us) service_pool.cc:225] Handling call
0426 13:21:07.254918 (+ 39172us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.302457 (+ 47539us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.307627 (+  5170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.299082 25099 rpcz_store.cc:276] 0426 13:21:05.216814 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.217211 (+   397us) service_pool.cc:225] Handling call
0426 13:21:05.234883 (+ 17672us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:05.344668 (+109785us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:05.344699 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:05.344738 (+    39us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:05.344779 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.261687 (+1916908us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540001ccd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.261758 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.261945 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2051,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":251815808}
W20240426 13:21:07.330358 24854 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 273) took 61669 ms (client timeout 15000 ms). Trace:
I20240426 13:21:07.345515 25098 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:07.324879 29630 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.327662 24852 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:07.346045 25023 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:21:07.348150 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.349269 30053 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.350127 25099 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20240426 13:21:07.350806 30053 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:21:07.351817 29761 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:07.351671 29871 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:07.352592 29761 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:07.353446 29790 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.728s	user 0.018s	sys 0.001s
I20240426 13:21:07.353670 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.354127 30073 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:07.354012 29790 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.351678 30033 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:07.355387 30033 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:07.356279 29989 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.356937 24953 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:21:07.356365 30033 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.357755 30033 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.356421 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.359992 30057 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:06.635363 24952 rpcz_store.cc:276] 0426 13:21:03.013137 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.643026 (+1629889us) service_pool.cc:225] Handling call
0426 13:21:04.643336 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:04.643565 (+   229us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:21:04.887561 (+243996us) log.cc:842] Serialized 14 byte log entry
0426 13:21:06.382327 (+1494766us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b1800553e40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb
0426 13:21:06.470220 (+ 87893us) log.cc:842] Serialized 14 byte log entry
0426 13:21:06.581771 (+111551us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:06.585038 (+  3267us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:06.585104 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.585217 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:06.585341 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":946,"prepare.queue_time_us":752139,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":3079,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":51020,"spinlock_wait_cycles":4394752,"thread_start_us":716103,"threads_started":4,"wal-append.queue_time_us":447329,"wal-append.run_cpu_time_us":1006,"wal-append.run_wall_time_us":1001037}
W20240426 13:21:07.351511 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 110768us
I20240426 13:21:07.365515 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.360085 29790 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.367046 30071 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:07.368467 30093 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.599s	user 0.000s	sys 0.004s
I20240426 13:21:07.308504 25026 rpcz_store.cc:276] 0426 13:21:04.873196 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.155367 (+2282171us) service_pool.cc:225] Handling call
0426 13:21:07.155778 (+   411us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.156445 (+   667us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:21:07.236564 (+ 80119us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.301676 (+ 65112us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.301723 (+    47us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:07.301766 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.301823 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.301944 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15170,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":1616,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":299,"thread_start_us":103578,"threads_started":2}
I20240426 13:21:07.372126 25105 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20240426 13:21:07.373919 25022 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:07.374897 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.374581 25026 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20240426 13:21:07.377234 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.354661 29761 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.375818 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 662) took 1916 ms. Trace:
I20240426 13:21:07.380543 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2732) took 1741 ms. Trace:
I20240426 13:21:07.383214 25029 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:07.317121 29759 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.387790 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.389837 25024 rpcz_store.cc:276] 0426 13:21:05.431905 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.311433 (+1879528us) service_pool.cc:225] Handling call
0426 13:21:07.311951 (+   518us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.345938 (+ 33987us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.347934 (+  1996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.389943 25026 rpcz_store.cc:276] 0426 13:21:05.638877 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.373510 (+1734633us) service_pool.cc:225] Handling call
0426 13:21:07.374339 (+   829us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:07.378102 (+  3763us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.380459 (+  2357us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.379755 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 665) took 1917 ms. Trace:
I20240426 13:21:07.390626 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2731) took 1745 ms. Trace:
I20240426 13:21:07.392416 25023 rpcz_store.cc:276] 0426 13:21:05.445686 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.323367 (+1877681us) service_pool.cc:225] Handling call
0426 13:21:07.323736 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.348489 (+ 24753us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.363312 (+ 14823us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.328500 30028 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:07.392042 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2734) took 1748 ms. Trace:
I20240426 13:21:07.394243 30028 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:07.393204 25022 rpcz_store.cc:276] 0426 13:21:05.634215 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.373248 (+1739033us) service_pool.cc:225] Handling call
0426 13:21:07.373566 (+   318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.376673 (+  3107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.379596 (+  2923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.400408 25029 rpcz_store.cc:276] 0426 13:21:05.643758 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.382170 (+1738412us) service_pool.cc:225] Handling call
0426 13:21:07.382621 (+   451us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.390289 (+  7668us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.391969 (+  1680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.410171 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.353408 29871 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.373013 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:07.409818 29761 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:07.415654 29761 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:07.415340 29871 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.418465 29871 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:07.424247 30079 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.037s	user 0.011s	sys 0.029s
W20240426 13:21:07.424443 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 118509us
I20240426 13:21:07.425371 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.426457 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 880) took 1259 ms. Trace:
I20240426 13:21:07.426998 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.426887 24949 rpcz_store.cc:276] 0426 13:21:06.167052 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.168068 (+  1016us) service_pool.cc:225] Handling call
0426 13:21:06.177099 (+  9031us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:06.177360 (+   261us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:06.177397 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:06.178599 (+  1202us) log.cc:842] Serialized 26 byte log entry
0426 13:21:06.359341 (+180742us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:06.359390 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.359455 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.423261 (+1063806us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:07.425975 (+  2714us) raft_consensus.cc:1671] finished
0426 13:21:07.425998 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.426263 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29664000,"thread_start_us":180269,"threads_started":1,"wal-append.queue_time_us":208355}
I20240426 13:21:07.433166 29894 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:07.433424 29446 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:07.433609 29894 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:07.426910 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.434752 25615 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11.
I20240426 13:21:07.435323 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 866) took 1610 ms. Trace:
I20240426 13:21:07.435619 25615 rpcz_store.cc:276] 0426 13:21:05.825215 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.825350 (+   135us) service_pool.cc:225] Handling call
0426 13:21:07.426553 (+1601203us) spinlock_profiling.cc:243] Waited 961 us on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:07.435225 (+  8672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2210048}
I20240426 13:21:07.434105 29894 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:07.434648 29759 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.440929 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.365886 29366 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.445950 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1984) took 1368 ms. Trace:
I20240426 13:21:07.446949 29366 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.447851 25103 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:07.448264 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.449663 25103 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20240426 13:21:07.451159 25103 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20240426 13:21:07.446473 29058 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.452236 30036 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.452749 30036 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.453985 25096 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:21:07.416117 29761 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:07.460178 25105 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20240426 13:21:07.461715 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 863) took 1659 ms. Trace:
I20240426 13:21:07.461827 25105 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:07.462169 25097 rpcz_store.cc:276] 0426 13:21:05.802252 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:05.802443 (+   191us) service_pool.cc:225] Handling call
0426 13:21:05.802738 (+   295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.051124 (+1248386us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.051313 (+   189us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.051512 (+   199us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:07.051706 (+   194us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.461335 (+409629us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.461392 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.461530 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44033,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"spinlock_wait_cycles":2864302848}
I20240426 13:21:07.467516 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 659) took 2036 ms. Trace:
I20240426 13:21:07.468418 24949 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20240426 13:21:07.469337 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1976) took 2389 ms. Trace:
I20240426 13:21:07.470132 30022 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20240426 13:21:07.471491 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106123us
W20240426 13:21:07.471935 30090 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.561s	user 0.000s	sys 0.002s
I20240426 13:21:07.470022 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.474233 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2622) took 1300 ms. Trace:
I20240426 13:21:07.469707 25031 rpcz_store.cc:276] 0426 13:21:05.080540 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.235750 (+2155210us) service_pool.cc:225] Handling call
0426 13:21:07.279469 (+ 43719us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:07.343600 (+ 64131us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.469093 (+125493us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.469282 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41020800}
I20240426 13:21:07.446375 25029 rpcz_store.cc:276] 0426 13:21:06.077624 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.411272 (+1333648us) service_pool.cc:225] Handling call
0426 13:21:07.445297 (+ 34025us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.445527 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.445556 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.445636 (+    80us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:07.445695 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.445764 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.445894 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.431138 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:07.478511 25023 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20240426 13:21:07.481593 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1980) took 1412 ms. Trace:
I20240426 13:21:07.482057 25023 rpcz_store.cc:276] 0426 13:21:06.069500 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.475485 (+1405985us) service_pool.cc:225] Handling call
0426 13:21:07.475850 (+   365us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.481263 (+  5413us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.481524 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.467927 25028 rpcz_store.cc:276] 0426 13:21:05.431333 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.250418 (+1819085us) service_pool.cc:225] Handling call
0426 13:21:07.466487 (+216069us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:21:07.466684 (+   197us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.467060 (+   376us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.467099 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.467184 (+    85us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:07.467238 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.467321 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.467450 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1362,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"spinlock_wait_cycles":29363456}
I20240426 13:21:07.474622 24948 rpcz_store.cc:276] 0426 13:21:06.173269 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.355264 (+181995us) service_pool.cc:225] Handling call
0426 13:21:06.355535 (+   271us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:06.355873 (+   338us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:06.355913 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:06.356406 (+   493us) log.cc:842] Serialized 26 byte log entry
0426 13:21:06.755182 (+398776us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:06.755227 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:06.755305 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.471046 (+715741us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:07.473744 (+  2698us) raft_consensus.cc:1671] finished
0426 13:21:07.473765 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.474030 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1802,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6383,"spinlock_wait_cycles":13821184,"thread_start_us":398446,"threads_started":1,"wal-append.queue_time_us":433795}
I20240426 13:21:07.472362 29386 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:07.508119 25102 raft_consensus.cc:2354] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12.
I20240426 13:21:07.508251 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.508950 29429 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.510210 25027 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:21:07.511013 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2727) took 1877 ms. Trace:
I20240426 13:21:07.511348 25027 rpcz_store.cc:276] 0426 13:21:05.633229 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.377132 (+1743903us) service_pool.cc:225] Handling call
0426 13:21:07.480274 (+103142us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.510674 (+ 30400us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.510943 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.508970 25030 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:07.512967 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1988) took 1406 ms. Trace:
I20240426 13:21:07.513301 25030 rpcz_store.cc:276] 0426 13:21:06.106758 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.478643 (+1371885us) service_pool.cc:225] Handling call
0426 13:21:07.508746 (+ 30103us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.512705 (+  3959us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.512910 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.515496 24956 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:07.509512 29429 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.517081 25029 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:21:07.517625 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1987) took 1411 ms. Trace:
I20240426 13:21:07.517902 29630 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:07.518134 29386 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.520845 29386 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.521003 29790 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.522033 29386 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:07.522769 29790 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:07.517930 25029 rpcz_store.cc:276] 0426 13:21:06.106511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.478517 (+1372006us) service_pool.cc:225] Handling call
0426 13:21:07.515175 (+ 36658us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.517432 (+  2257us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.517584 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.525954 30036 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.529464 25031 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:07.548849 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1986) took 1442 ms. Trace:
I20240426 13:21:07.550328 25031 rpcz_store.cc:276] 0426 13:21:06.106236 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.478202 (+1371966us) service_pool.cc:225] Handling call
0426 13:21:07.527689 (+ 49487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.548510 (+ 20821us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.548774 (+   264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.551837 25101 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:07.533772 25029 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20240426 13:21:07.553558 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.548611 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.554808 25105 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:07.515926 24956 pending_rounds.cc:76] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 7
I20240426 13:21:07.529286 29058 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.529248 29790 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.563580 25103 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
W20240426 13:21:07.561319 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2336) took 3106 ms (client timeout 1911 ms). Trace:
I20240426 13:21:07.518304 29688 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:07.566675 25023 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20240426 13:21:07.568473 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 671) took 1247 ms. Trace:
I20240426 13:21:07.568791 25023 rpcz_store.cc:276] 0426 13:21:06.320876 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.504702 (+1183826us) service_pool.cc:225] Handling call
0426 13:21:07.555968 (+ 51266us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.568284 (+ 12316us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.568436 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.565235 24956 pending_rounds.cc:97] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:07.564054 25103 pending_rounds.cc:76] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 6
I20240426 13:21:07.570173 25103 pending_rounds.cc:97] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 11.7
W20240426 13:21:07.564543 25102 rpcz_store.cc:269] 0426 13:21:04.454473 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.454634 (+   161us) service_pool.cc:225] Handling call
0426 13:21:07.561065 (+3106431us) spinlock_profiling.cc:243] Waited 3 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.561241 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6898151680}
I20240426 13:21:07.551335 29630 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20240426 13:21:07.573448 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 872) took 1224 ms. Trace:
I20240426 13:21:07.573837 25624 rpcz_store.cc:276] 0426 13:21:06.349057 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.349212 (+   155us) service_pool.cc:225] Handling call
0426 13:21:06.359339 (+ 10127us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:06.359521 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:06.711819 (+352298us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.572813 (+860994us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.572900 (+    87us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:07.572987 (+    87us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.573093 (+   106us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.573268 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":909843,"prepare.queue_time_us":410260,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":87,"thread_start_us":345003,"threads_started":1}
I20240426 13:21:07.575619 25025 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20240426 13:21:07.573637 30113 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20240426 13:21:07.576356 29429 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.575845 25023 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20240426 13:21:07.576900 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1985) took 1471 ms. Trace:
I20240426 13:21:07.563066 29386 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20240426 13:21:07.577271 25025 rpcz_store.cc:276] 0426 13:21:06.105918 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.427235 (+1321317us) service_pool.cc:225] Handling call
0426 13:21:07.480332 (+ 53097us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.575963 (+ 95631us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.576721 (+   758us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b54002add50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:07.576845 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117060992}
I20240426 13:21:07.578997 25023 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:07.569519 24956 raft_consensus.cc:2866] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:07.580565 29759 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:07.581012 29759 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:07.581423 29759 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.581197 29058 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.583108 29058 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.584690 25029 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20240426 13:21:07.586197 30011 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:07.586583 30011 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.570504 25103 raft_consensus.cc:2866] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:07.587483 30011 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.597074 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:07.597510 30011 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:07.597420 30072 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:07.598655 29872 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:07.599218 30072 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:07.602859 25026 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:07.603464 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 669) took 1789 ms. Trace:
I20240426 13:21:07.603775 25026 rpcz_store.cc:276] 0426 13:21:05.814208 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.548703 (+1734495us) service_pool.cc:225] Handling call
0426 13:21:07.568988 (+ 20285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.603228 (+ 34240us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.603407 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.606009 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.559888 30073 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:07.600411 29872 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.612514 25030 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:07.615351 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1992) took 1204 ms. Trace:
I20240426 13:21:07.615733 25030 rpcz_store.cc:276] 0426 13:21:06.411177 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.528472 (+1117295us) service_pool.cc:225] Handling call
0426 13:21:07.566442 (+ 37970us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.615100 (+ 48658us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.615292 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.621621 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.624289 25028 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
W20240426 13:21:07.629506 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 105226us
I20240426 13:21:07.630839 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.631891 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.629091 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1990) took 1218 ms. Trace:
I20240426 13:21:07.635324 25028 rpcz_store.cc:276] 0426 13:21:06.410581 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.507298 (+1096717us) service_pool.cc:225] Handling call
0426 13:21:07.566401 (+ 59103us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.624750 (+ 58349us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.629036 (+  4286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:07.643287 30049 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:07.643769 30049 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:07.644165 30049 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:07.644735 25101 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20240426 13:21:07.645857 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:07.646528 25097 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:21:07.648520 29872 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.648545 29761 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.640528 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:07.657809 25102 raft_consensus.cc:2396] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20240426 13:21:07.660022 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:07.663928 25029 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:07.665133 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57174.7us
I20240426 13:21:07.668499 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:07.670572 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.672693 25097 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:07.675832 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.676242 25028 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:07.677114 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:07.568075 29688 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:07.683151 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 178504us
I20240426 13:21:07.683874 25099 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.693751 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.694339 25029 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:07.696892 25105 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:21:07.701179 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.702762 25025 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:07.703653 24948 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:07.704851 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.707139 25030 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:07.708436 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.708911 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:07.709270 25029 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:07.717868 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:07.718905 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:07.717278 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.715072 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.703469 30072 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.672097 29790 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.697396 29894 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.705708 29954 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.722019 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:07.715015 29761 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.726953 29630 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:07.728914 25618 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20240426 13:21:07.717609 30033 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:07.721531 29938 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:07.346145 24854 rpcz_store.cc:269] 0426 13:20:05.661002 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.668986 (+  7984us) service_pool.cc:225] Handling call
0426 13:21:07.330041 (+61661055us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:57.947943 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:58.063715 (+115772us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:58.063768 (+    53us) write_op.cc:427] Acquired schema lock
0426 13:20:58.063780 (+    12us) tablet.cc:660] Decoding operations
0426 13:20:58.181224 (+117444us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:58.189836 (+  8612us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:58.194116 (+  4280us) tablet.cc:683] Acquiring locks for 3 operations
0426 13:20:58.221348 (+ 27232us) tablet.cc:699] Row locks acquired
0426 13:20:58.221379 (+    31us) write_op.cc:260] PREPARE: finished
0426 13:20:58.221555 (+   176us) write_op.cc:270] Start()
0426 13:20:58.221693 (+   138us) write_op.cc:275] Timestamp: P: 1714137658221482 usec, L: 0
0426 13:20:58.224480 (+  2787us) op_driver.cc:331] REPLICATION: starting
0426 13:20:58.293835 (+ 69355us) log.cc:842] Serialized 1299 byte log entry
0426 13:20:58.792111 (+498276us) op_driver.cc:444] REPLICATION: finished
0426 13:20:59.128390 (+336279us) write_op.cc:300] APPLY: starting
0426 13:20:59.128550 (+   160us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:59.131131 (+  2581us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:59.131243 (+   112us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:59.298232 (+166989us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:59.298537 (+   305us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0426 13:20:59.298595 (+    58us) write_op.cc:311] APPLY: finished
0426 13:20:59.312156 (+ 13561us) log.cc:842] Serialized 41 byte log entry
0426 13:20:59.312322 (+   166us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:59.312414 (+    92us) write_op.cc:446] Released schema lock
0426 13:20:59.312466 (+    52us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:59.312799 (+   333us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:59.312828 (+    29us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":453101,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":288637,"apply.run_cpu_time_us":7319,"apply.run_wall_time_us":184758,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16673,"num_ops":3,"prepare.queue_time_us":421070,"prepare.run_cpu_time_us":11588,"prepare.run_wall_time_us":500404,"raft.queue_time_us":141335,"raft.run_cpu_time_us":65581,"raft.run_wall_time_us":238890,"replication_time_us":544650,"thread_start_us":435252,"threads_started":4,"wal-append.queue_time_us":154368,"wal-append.run_cpu_time_us":23937,"wal-append.run_wall_time_us":1837289}]]}
I20240426 13:21:07.756395 25624 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:07.737372 29630 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.773828 29630 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:07.784066 25069 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11.
I20240426 13:21:07.784538 25069 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:07.785039 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 642) took 1434 ms. Trace:
W20240426 13:21:07.785282 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 150212us
I20240426 13:21:07.785943 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.785473 25622 rpcz_store.cc:276] 0426 13:21:06.350256 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.350376 (+   120us) service_pool.cc:225] Handling call
0426 13:21:06.350661 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:06.350873 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:21:06.453671 (+102798us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.766869 (+1313198us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.784346 (+ 17477us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.784430 (+    84us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:07.784558 (+   128us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.784663 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.784859 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":861146,"prepare.queue_time_us":206813,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":824388,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":524435,"threads_started":3,"wal-append.queue_time_us":457893}
I20240426 13:21:07.787472 29557 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:07.785472 29560 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:21:07.788048 29557 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.789187 29938 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.791713 29938 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.795099 29560 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11.
I20240426 13:21:07.795573 30029 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:07.717446 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:07.801023 25097 raft_consensus.cc:595] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.801445 25097 raft_consensus.cc:603] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:07.786813 29429 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:21:07.807862 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 108378us
I20240426 13:21:07.817174 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.785547 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.747383 30072 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.824265 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:07.825459 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 141951us
I20240426 13:21:07.725230 29688 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.756559 29954 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:07.830392 30118 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:21:07.837630 25622 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:07.833882 29688 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:07.849606 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.852542 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:07.863469 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:07.865530 29909 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:07.865875 30119 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20240426 13:21:07.854866 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1982) took 1780 ms. Trace:
I20240426 13:21:07.866050 29909 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.850522 29058 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:07.841044 29630 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.878850 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 667) took 2174 ms. Trace:
I20240426 13:21:07.879415 29909 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:07.879297 25024 rpcz_store.cc:276] 0426 13:21:05.704769 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.420685 (+1715916us) service_pool.cc:225] Handling call
0426 13:21:07.421133 (+   448us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.421702 (+   569us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.421741 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:07.422240 (+   499us) log.cc:842] Serialized 26 byte log entry
0426 13:21:07.665479 (+243239us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:07.665531 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.666868 (+  1337us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.876082 (+209214us) raft_consensus.cc:1671] finished
0426 13:21:07.876102 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.878780 (+  2678us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":179,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":242870,"threads_started":1,"wal-append.queue_time_us":290700}
I20240426 13:21:07.880796 29759 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.882721 25027 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
W20240426 13:21:07.884562 30117 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20240426 13:21:07.885373 29761 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:07.885953 29761 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:07.886240 29956 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:07.886696 29956 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.886833 29761 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.888494 25616 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:07.888864 25616 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:07.889554 25616 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.890610 25616 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:07.867866 25022 rpcz_store.cc:276] 0426 13:21:06.070218 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.430106 (+1359888us) service_pool.cc:225] Handling call
0426 13:21:07.461980 (+ 31874us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:21:07.462534 (+   554us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.462566 (+    32us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:21:07.463436 (+   870us) log.cc:842] Serialized 78 byte log entry
0426 13:21:07.666485 (+203049us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:07.666533 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.666617 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.850297 (+183680us) raft_consensus.cc:1671] finished
0426 13:21:07.850315 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.850440 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5647,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":202607,"threads_started":1,"wal-append.queue_time_us":305071}
I20240426 13:21:07.895767 25026 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
W20240426 13:21:07.870561 30120 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:21:07.898028 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.866145 29944 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:07.901360 29909 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:07.901834 29909 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.904551 29557 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.905427 24947 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:07.896415 29894 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.828308 30028 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.866513 30022 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:07.896652 29934 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:07.911320 25616 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20240426 13:21:07.883968 29759 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.914057 25616 pending_rounds.cc:76] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 11
I20240426 13:21:07.909911 29058 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.920403 25616 pending_rounds.cc:97] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20240426 13:21:07.921056 25616 raft_consensus.cc:2866] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:07.921456 30022 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.945313 30022 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.922044 30125 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:07.945878 30066 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:07.946156 30125 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:07.948138 30125 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.887540 29956 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:07.922847 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.949131 29956 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:07.949643 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.950021 29058 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.952945 24951 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:07.913914 29934 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:07.954615 29934 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:07.918272 30122 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20240426 13:21:07.913576 30028 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:07.955840 29934 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:07.956493 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.957554 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:07.958187 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.922874 29954 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:07.959925 30120 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240426 13:21:07.929302 30132 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20240426 13:21:07.930156 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1991) took 1519 ms. Trace:
W20240426 13:21:07.961927 30135 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:21:07.961817 25027 rpcz_store.cc:276] 0426 13:21:06.410904 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.528401 (+1117497us) service_pool.cc:225] Handling call
0426 13:21:07.562414 (+ 34013us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:07.883181 (+320767us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.929928 (+ 46747us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54002fa154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:07.930103 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":736094592}
I20240426 13:21:07.963356 24956 raft_consensus.cc:1203] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.10->[13.11-15.12]   Dedup: 13.11->[15.12-15.12]
I20240426 13:21:07.963838 24956 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: [13.11-15.12]
I20240426 13:21:07.964705 29909 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:07.965037 30066 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:07.930931 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.906471 29944 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:07.932003 30072 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:07.932507 30115 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.254s	user 0.007s	sys 0.000s
W20240426 13:21:07.934787 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 228975us
I20240426 13:21:07.968267 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1965) took 1053 ms. Trace:
I20240426 13:21:07.936947 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:07.937670 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93342.9us
I20240426 13:21:07.939303 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:07.939697 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 874) took 1525 ms. Trace:
I20240426 13:21:07.940333 29630 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:07.941504 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:07.948320 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100384us
I20240426 13:21:07.925652 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.976807 25099 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:07.968712 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:07.957581 25590 connection.cc:232] RPC call timeout handler was delayed by -1.4446108880000565s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20240426 13:21:07.980387 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:07.981760 25096 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:07.984457 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:07.983736 25028 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:07.985603 24952 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:07.968580 25100 rpcz_store.cc:276] 0426 13:21:06.915003 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.380985 (+465982us) service_pool.cc:225] Handling call
0426 13:21:07.381502 (+   517us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:07.382081 (+   579us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:07.382118 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:07.382723 (+   605us) log.cc:842] Serialized 26 byte log entry
0426 13:21:07.505498 (+122775us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:07.505554 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.505637 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:07.932232 (+426595us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b240024d898. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d247ce4 00007fda8d245dfb 00007fda8d2d3e09 00007fda8d2d3a95 00007fda8d2d30dd 00007fda8ff822ee 00007fda8ff99a8c 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1
0426 13:21:07.967884 (+ 35652us) raft_consensus.cc:1671] finished
0426 13:21:07.967904 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.968118 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1996,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"spinlock_wait_cycles":7100288,"thread_start_us":122152,"threads_started":1,"wal-append.queue_time_us":235571}
I20240426 13:21:07.984849 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.031648 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.036525 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.971061 25621 rpcz_store.cc:276] 0426 13:21:06.349647 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.349787 (+   140us) service_pool.cc:225] Handling call
0426 13:21:06.350038 (+   251us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:06.350260 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:06.454409 (+104149us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.509791 (+1055382us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b180047b300. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:21:07.723701 (+213910us) log.cc:842] Serialized 14 byte log entry
0426 13:21:07.848328 (+124627us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:07.854207 (+  5879us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:07.869283 (+ 15076us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:07.869371 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:07.874809 (+  5438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116396,"prepare.queue_time_us":281340,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":11725,"raft.queue_time_us":1141626,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":5089,"spinlock_wait_cycles":327695232,"thread_start_us":289601,"threads_started":3,"wal-append.queue_time_us":190968}
I20240426 13:21:08.039805 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:07.987818 30022 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:08.023280 29446 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:08.020169 29909 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.029253 29790 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:08.054772 29666 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.056573 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.057785 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:08.053802 30022 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.060618 30022 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:08.068269 29666 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.068954 29446 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:08.070287 29446 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.071005 29666 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:08.136633 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106836us
I20240426 13:21:08.096850 29909 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.022244 29058 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20240426 13:21:08.173918 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.176079 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:08.191602 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.175954 30021 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:08.194185 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 169409us
W20240426 13:21:08.201767 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 167424us
W20240426 13:21:08.188184 30136 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.142s	user 0.004s	sys 0.000s
W20240426 13:21:08.185668 30049 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.007s
I20240426 13:21:08.207198 30056 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:08.207607 30056 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:08.207991 30056 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:08.210625 29685 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:21:08.212563 30137 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20240426 13:21:08.210996 29685 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:08.213291 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.214061 30011 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:08.214558 30011 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:08.215126 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:08.216253 29956 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:08.216403 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.216753 29956 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:08.217715 29956 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.215421 30011 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.190116 29630 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:08.088877 29944 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.262193 29685 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.257174 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.266436 29685 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:08.274749 29560 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:08.229391 30049 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.279886 29944 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:08.298557 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83494.8us
I20240426 13:21:08.301038 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.267694 29630 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.304091 24955 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:08.292515 29688 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20240426 13:21:08.327661 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 141479us
I20240426 13:21:08.320220 29761 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:08.294340 29560 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.336241 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.335906 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:08.355934 25622 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.000s
I20240426 13:21:08.357144 25622 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:08.362768 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 377993us
I20240426 13:21:08.332099 29872 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.018s
I20240426 13:21:08.345786 29761 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:08.348215 29909 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.335943 30072 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:21:08.370690 29643 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:08.371538 30146 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:21:08.371587 29560 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.376137 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.343252 29871 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20240426 13:21:08.359405 29894 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.368459 29761 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:08.378621 25104 raft_consensus.cc:2396] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20240426 13:21:08.374691 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:08.348701 30022 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.380126 25097 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:08.364643 29994 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.383790 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.384199 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.386986 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.388176 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:08.388278 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.384907 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.397440 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.404594 29386 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:08.405620 29872 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:08.407043 29386 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:08.407997 29386 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:08.408366 30073 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:08.408380 29386 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:08.408962 30073 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:08.409714 29386 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:08.410538 30141 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20240426 13:21:08.411432 29386 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:08.413125 30056 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.414719 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.415768 25105 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:08.407073 29775 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:08.409760 30073 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.419093 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:08.408212 30139 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20240426 13:21:08.423053 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:08.426074 30142 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20240426 13:21:08.379935 29894 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:08.421995 30036 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.394460 29994 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.398387 30125 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:08.440644 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:08.437844 30036 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.432412 29894 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:08.449752 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:08.450992 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:08.448855 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:08.378839 29643 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.432890 30049 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:08.455549 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.443307 30152 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:08.444756 30073 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:08.458055 25097 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:08.476684 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.479660 24950 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20240426 13:21:08.455034 30138 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
W20240426 13:21:08.458889 29934 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.003s
I20240426 13:21:08.474783 29871 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.485021 29871 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.486845 29871 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:08.444339 29994 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.458631 30125 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:08.489672 30125 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:08.482906 29759 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.487802 30022 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.490705 29759 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.457870 30056 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.475147 29630 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:08.498461 29871 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20240426 13:21:08.482837 29560 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:08.506752 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:08.508975 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 84549.8us
W20240426 13:21:08.509725 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133272us
I20240426 13:21:08.490667 30125 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.522719 25624 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:08.544238 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:08.560117 30148 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:08.561059 30154 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.561609 30154 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.562026 30154 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:08.565130 30151 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
W20240426 13:21:08.566879 30147 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20240426 13:21:08.570006 30150 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20240426 13:21:08.574486 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51054.5us
I20240426 13:21:08.520990 30036 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:08.601553 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:08.603163 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:08.605157 30148 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:08.605710 30148 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:08.610152 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89387.7us
I20240426 13:21:08.611040 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:08.613173 30029 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:08.613742 30029 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:08.616400 29446 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.616775 29643 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:08.616935 29446 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.618623 29759 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:08.619029 29446 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:08.621240 29944 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:08.622519 29944 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:08.623304 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.622584 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:08.619875 30145 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20240426 13:21:08.624519 24948 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
W20240426 13:21:08.628494 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 223197us
I20240426 13:21:08.640573 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:08.642242 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 162145us
I20240426 13:21:08.645501 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:08.646780 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:08.648638 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.648896 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.649991 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.670459 25617 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:08.688589 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:08.689253 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44295.6us
I20240426 13:21:08.692636 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.671413 30149 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.690696 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.756743 30149 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:08.757133 30149 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:08.757195 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66590.8us
I20240426 13:21:08.687947 30155 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:08.760963 30148 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:08.695053 29759 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.761534 30155 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:08.761853 29759 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.762425 30056 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:08.762743 30056 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:08.763520 30056 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:08.761373 30148 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:08.764906 30148 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:08.698016 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49933.7us
W20240426 13:21:08.768155 30132 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20240426 13:21:08.698055 29058 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:08.701233 29685 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:08.704500 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.705415 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.712229 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.779353 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:08.712914 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.713704 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:08.672415 30036 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:08.742532 30021 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.002s
I20240426 13:21:08.756110 29446 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.786623 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:08.805243 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109850us
I20240426 13:21:08.805747 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:08.692090 29688 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:08.814281 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121734us
I20240426 13:21:08.819131 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:08.822786 25101 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20240426 13:21:08.682417 30048 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.124s
I20240426 13:21:08.806396 30021 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:08.804775 29446 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.819000 29994 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:08.857201 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51322.2us
I20240426 13:21:08.858180 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:08.824108 29871 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.823588 30166 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:08.782081 29894 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.827517 29666 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:08.823802 29688 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.861169 29446 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:08.867890 29871 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:08.899580 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:08.909631 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.910202 24950 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:08.838496 29630 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20240426 13:21:08.861449 29994 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:08.839542 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.838948 29909 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:08.917661 25617 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20240426 13:21:08.868944 29934 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20240426 13:21:08.919759 24948 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
W20240426 13:21:08.929487 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 110428us
I20240426 13:21:08.927448 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.930827 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:08.932144 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 154365us
I20240426 13:21:08.918577 29994 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:08.877161 29761 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.913452 29871 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:08.954073 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:08.875496 30166 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:08.926756 29058 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:08.925149 30167 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.924674 29790 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20240426 13:21:08.950542 30048 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:08.962401 30163 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:08.967258 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:08.952860 30011 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:08.971398 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:08.929279 29775 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20240426 13:21:08.972786 24950 raft_consensus.cc:2396] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20240426 13:21:08.964232 29560 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:08.974659 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:08.945050 30164 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20240426 13:21:08.976577 30165 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:08.977197 30172 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20240426 13:21:08.977826 30168 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:21:08.966764 30029 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:08.952508 29909 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20240426 13:21:08.966774 29759 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:08.969458 30033 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:09.006968 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 44090us
I20240426 13:21:08.988188 29560 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.013509 29560 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:09.013927 30056 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:09.010535 30154 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:09.017093 30171 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20240426 13:21:08.988442 30049 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:09.008157 29759 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.005522 30011 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:09.031383 30169 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20240426 13:21:09.031833 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.032867 30166 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.056561 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.023876 29688 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:08.970798 30167 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.060966 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:08.979982 30163 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.025796 30021 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.026271 29894 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:09.090788 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53006.3us
W20240426 13:21:09.091239 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53267us
W20240426 13:21:09.092273 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86620.2us
I20240426 13:21:08.991324 30165 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.008339 30033 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.108512 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.108512 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:09.093329 30166 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:09.114079 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 203575us
I20240426 13:21:09.115607 25028 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:09.116107 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:09.121436 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.099925 29761 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.124128 25101 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20240426 13:21:09.125929 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.094380 30164 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20240426 13:21:09.089547 29759 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.037863 29058 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.106053 30190 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.130003 25027 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:09.112257 30179 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.113451 30049 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.138628 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.140679 30033 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.122074 30182 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.159039 25024 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20240426 13:21:09.155802 25028 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:09.132673 29894 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.161955 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:09.130672 29761 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.166785 25031 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20240426 13:21:09.162302 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.117790 30021 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.170075 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.163914 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:09.173578 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52095.7us
I20240426 13:21:09.174707 25022 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:09.175052 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:09.123466 30166 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.156740 29688 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.181962 24955 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20240426 13:21:09.172189 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:09.186793 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87974.3us
I20240426 13:21:09.164768 30154 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.187955 25615 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20240426 13:21:09.188961 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:09.190119 30183 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20240426 13:21:09.191658 29759 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.193792 29643 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:09.195902 30163 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.196938 29956 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:09.197309 29956 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:09.197867 30176 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
W20240426 13:21:09.199450 30170 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20240426 13:21:09.199776 30165 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.143342 30190 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:09.201508 30177 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
I20240426 13:21:09.201836 30190 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.131815 30188 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.053s
I20240426 13:21:09.186460 25027 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:21:09.221973 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47028.5us
I20240426 13:21:09.222771 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:09.208237 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.144639 30179 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.237746 30201 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.238076 30199 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.238337 30154 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:09.238601 30199 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.238981 30154 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.239051 30179 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.239547 30154 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.240736 30199 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.241189 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.241482 30154 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.242388 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.242583 30021 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.174052 29761 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.243324 30154 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.244144 30195 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240426 13:21:09.245150 29994 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.245816 25105 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:09.237753 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.243553 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:09.244465 29761 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.237993 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.254093 25617 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:09.179603 30182 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.184921 29560 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.264416 29560 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.264823 30164 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:09.255337 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.144464 30011 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.256994 25100 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:09.266491 25618 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:09.258692 25101 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:09.176369 29894 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.234464 30049 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.281260 30166 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.186339 30149 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.282359 25102 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:09.282233 29446 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.262838 30182 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.265226 30164 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.258147 25096 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:09.269740 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80827.3us
W20240426 13:21:09.274479 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51808.3us
I20240426 13:21:09.246696 29386 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:21:09.238283 30201 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.210405 30148 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.276836 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:09.277029 29994 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.273443 29761 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.278842 29871 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.279542 29894 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.279872 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:09.234196 29688 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.280140 30195 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:09.263962 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:09.261299 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.243779 30154 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:09.291936 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 83765.3us
I20240426 13:21:09.293706 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.291721 25022 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:09.294238 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.295190 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.297530 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.298556 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.296588 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:09.234652 30056 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.054s
I20240426 13:21:09.276463 30167 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:09.303658 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 67966.9us
I20240426 13:21:09.332772 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.304282 30193 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.304759 30197 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20240426 13:21:09.287825 29630 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.309494 25105 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20240426 13:21:09.210815 30192 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20240426 13:21:09.297152 30190 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:09.299899 29386 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:09.326606 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.327332 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.328639 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.357437 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.304325 30185 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:09.331295 25031 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:09.331672 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.340436 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.350708 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.362972 25619 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:09.362315 25615 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
W20240426 13:21:09.352449 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55320.4us
I20240426 13:21:09.356408 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.331835 30149 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.366655 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.320931 29761 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.341902 29894 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:09.371209 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.371282 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.360992 29386 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:09.381281 25022 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:09.381999 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.382577 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.356524 29630 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.409726 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.411428 25022 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:09.379966 29446 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.379115 30163 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:09.343072 30193 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.402336 30190 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:09.379841 30191 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:09.417168 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.408430 30198 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.398169 30207 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:09.414883 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.408174 30182 raft_consensus.cc:2741] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20240426 13:21:09.356729 30164 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.416788 29759 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:09.423635 30195 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.422777 30198 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.426671 30193 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.426870 30195 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.418009 30190 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:09.434010 30201 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.435148 30154 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:09.435776 30148 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.441514 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.441612 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.442150 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:09.442953 30011 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.443219 25022 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:09.443672 30011 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.446331 30011 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.447824 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.460351 25617 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:09.422611 30189 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:09.422732 30163 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:09.452132 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 72483.4us
I20240426 13:21:09.432965 30190 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:09.472462 29894 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:09.474365 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.461140 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:09.465631 30206 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20240426 13:21:09.475299 24952 raft_consensus.cc:2396] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20240426 13:21:09.475198 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.476022 24949 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:09.427944 30198 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.476927 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.467232 29386 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.474546 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.479051 29386 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.468322 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.478981 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.479897 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.480856 24954 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20240426 13:21:09.477587 24955 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:09.452232 29871 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.482306 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.484421 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.491433 24956 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:09.491638 29630 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.492766 30195 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.496819 29761 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.496897 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.482745 30207 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20240426 13:21:09.462507 29688 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.499863 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.501013 25027 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20240426 13:21:09.505472 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.506103 25621 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:09.506942 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.511150 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.513280 25617 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:09.514406 25620 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:09.512773 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.516633 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.531944 24949 raft_consensus.cc:2396] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:21:09.532886 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:09.516557 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55582.8us
I20240426 13:21:09.534476 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.536358 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.515803 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.517722 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.540436 25617 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:09.543422 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.514808 30049 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:09.543795 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:09.545274 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.516248 30189 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:09.551678 30163 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:09.554123 29759 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:09.554581 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.555811 30165 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:09.556290 30165 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:09.557134 30165 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.542934 30033 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:09.544947 30011 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:09.567600 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:09.570103 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.571465 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.550689 30182 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:09.549922 29894 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:09.562466 30210 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:09.565881 30217 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:09.593222 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.593135 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:09.610742 25030 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:09.564136 30049 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:09.587891 29871 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:09.590166 29446 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:09.591002 30163 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:09.592803 30166 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:09.578473 30182 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:09.628350 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:09.627980 30211 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:09.593575 30021 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.593602 29630 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20240426 13:21:09.626539 29934 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20240426 13:21:09.611554 30190 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.633275 30232 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:09.683073 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:09.611690 30217 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.625370 29446 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:09.620579 30049 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:09.591914 30011 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20240426 13:21:09.667264 30021 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.674979 30190 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.675580 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.686843 25617 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:09.680790 30221 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20240426 13:21:09.681842 30228 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:21:09.696864 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.682530 30189 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.706372 30190 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:09.685866 30217 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:21:09.710407 30173 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.756s	user 0.003s	sys 0.000s
I20240426 13:21:09.711720 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2155) took 1136 ms. Trace:
I20240426 13:21:09.711798 25030 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:09.682915 30232 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.686005 29446 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:09.724169 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131099us
I20240426 13:21:09.687489 30049 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:09.734076 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:09.740831 29895 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:09.732197 30163 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.741979 30163 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:09.729619 30190 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.745721 30190 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.747188 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:09.747721 30190 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:09.748420 30209 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.368s	user 0.000s	sys 0.003s
I20240426 13:21:09.751488 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:09.752563 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 158542us
I20240426 13:21:09.712111 24953 rpcz_store.cc:276] 0426 13:21:08.574840 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:08.575046 (+   206us) service_pool.cc:225] Handling call
0426 13:21:08.606857 (+ 31811us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:08.607296 (+   439us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:08.607324 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:08.607821 (+   497us) log.cc:842] Serialized 26 byte log entry
0426 13:21:08.783698 (+175877us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:08.783759 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:08.783841 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:09.709831 (+925990us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:09.711446 (+  1615us) raft_consensus.cc:1671] finished
0426 13:21:09.711466 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:09.711666 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"spinlock_wait_cycles":5227392,"thread_start_us":175437,"threads_started":1,"wal-append.queue_time_us":309473}
I20240426 13:21:09.764896 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.764853 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.767167 29666 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:09.590265 30199 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.175s
I20240426 13:21:09.771332 29386 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:09.772787 29666 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.774760 29666 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:09.774926 30231 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:21:09.769734 30048 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:09.776895 30048 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.777267 30048 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:09.772917 29994 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.007s
I20240426 13:21:09.764966 29895 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:09.783583 30213 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:09.783978 30213 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.784320 30232 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.784722 30213 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.785395 30232 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:09.785694 30213 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:09.789449 30220 consensus_queue.cc:1159] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 12 status { last_received { term: 11 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:09.790601 30226 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:09.791065 30226 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.790721 29446 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.792995 30226 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.794024 30226 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:09.794379 25031 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:09.794728 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.768783 30165 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.796393 30165 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.796892 30163 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:09.797480 29994 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:09.797588 29956 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.798445 30165 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.798602 29994 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:09.799052 29994 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:09.799417 30200 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.381s	user 0.002s	sys 0.000s
I20240426 13:21:09.799896 29994 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.801790 29386 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:09.802047 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 880) took 1099 ms. Trace:
I20240426 13:21:09.802472 25102 rpcz_store.cc:276] 0426 13:21:08.702630 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:08.707333 (+  4703us) service_pool.cc:225] Handling call
0426 13:21:08.707694 (+   361us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:09.282866 (+575172us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:09.801851 (+518985us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:09.801998 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1321147776}
I20240426 13:21:09.802822 29386 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.804224 29994 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:09.807660 25617 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:09.808743 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.810652 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:09.798636 29956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:09.812012 25026 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20240426 13:21:09.813433 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:09.821122 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.806299 29446 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:09.727371 30182 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.826704 29666 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.812338 30190 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:09.832445 30191 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:09.822973 30237 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:21:09.841004 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.812316 30165 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.851617 29666 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.841233 30237 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:09.854388 30237 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:21:09.855438 30203 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.455s	user 0.003s	sys 0.000s
I20240426 13:21:09.856760 29790 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:09.847455 30190 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:09.841773 29630 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:09.831804 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:09.857172 29790 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:09.917253 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 617973us
I20240426 13:21:09.947497 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.874857 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:09.869352 29630 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:09.847997 30021 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:09.924551 30167 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:09.926796 25104 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20240426 13:21:09.927184 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 582791us
I20240426 13:21:09.927832 30185 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:09.928961 29685 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
W20240426 13:21:09.941345 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 397597us
W20240426 13:21:09.951439 30238 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:21:09.951386 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.949102 25028 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:09.862337 30165 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.974064 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:21:09.960924 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 119990us
I20240426 13:21:09.881255 30190 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:09.970260 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.972472 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:09.972910 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.976326 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.952226 29790 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:09.976774 24948 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:21:09.975733 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.955909 29630 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:09.980320 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.980870 24948 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:09.946303 30237 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:09.961534 30204 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:09.986056 25623 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:09.992583 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:09.981844 30179 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20240426 13:21:09.981938 30187 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:09.997328 30187 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:09.997656 30187 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:09.970952 30191 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:10.001936 25028 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:10.004653 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:09.984920 30185 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:10.008502 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:10.008121 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:10.008742 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:10.012104 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:09.987865 30204 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:10.014063 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:09.963754 30167 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.022578 24948 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:09.956447 30048 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.029075 25022 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20240426 13:21:10.035375 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:09.997803 30049 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.013059 30185 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:09.995401 29775 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20240426 13:21:10.012784 30154 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:10.019315 30204 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:10.017938 29909 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:10.065012 24951 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:10.067865 24952 raft_consensus.cc:2354] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:10.065382 24951 pending_rounds.cc:76] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 7
I20240426 13:21:10.069141 24951 pending_rounds.cc:97] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:10.069451 24951 raft_consensus.cc:2866] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:10.005906 30217 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.047495 30185 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:10.050729 30152 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:10.029090 30178 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20240426 13:21:10.102524 30246 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:21:10.183236 30237 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.226014 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.226627 25022 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:10.187700 25620 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:10.189453 30164 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:10.189594 30250 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.116s	user 0.002s	sys 0.007s
I20240426 13:21:10.090745 30191 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.072901 29909 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:10.091418 30152 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:10.072808 29429 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:10.190001 30252 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.107s	user 0.003s	sys 0.006s
W20240426 13:21:10.197950 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 168338us
I20240426 13:21:10.198282 29994 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:10.199371 30197 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:10.199659 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.294682 25104 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:10.200400 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:10.201267 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 233888us
I20240426 13:21:10.201323 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.205793 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 683) took 1007 ms. Trace:
I20240426 13:21:10.205729 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.296494 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:10.206524 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.207934 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:10.208230 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 135563us
I20240426 13:21:10.300961 25103 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:10.208490 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.209084 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.210840 30187 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.211501 25105 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:10.219691 30148 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:10.233960 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.233877 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.242338 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:10.233278 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:10.223680 24950 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:10.254299 30167 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:10.255632 30048 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:10.255473 29630 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.339823 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.260234 24951 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:21:10.309587 25096 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:10.296290 25030 rpcz_store.cc:276] 0426 13:21:09.198086 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.780907 (+582821us) service_pool.cc:225] Handling call
0426 13:21:09.781276 (+   369us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:09.781596 (+   320us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:09.781624 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:09.782065 (+   441us) log.cc:842] Serialized 52 byte log entry
0426 13:21:09.999590 (+217525us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:10.023658 (+ 24068us) log.cc:842] Serialized 14 byte log entry
0426 13:21:10.202179 (+178521us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:10.202254 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:10.205565 (+  3311us) raft_consensus.cc:1671] finished
0426 13:21:10.205599 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:10.205736 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":314639,"prepare.queue_time_us":23885,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":196011,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":124,"thread_start_us":217211,"threads_started":1,"wal-append.queue_time_us":302645}
W20240426 13:21:10.323832 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 81577.8us
I20240426 13:21:10.332067 25620 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:21:10.345999 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:10.346565 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 113408us
I20240426 13:21:10.347673 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.350456 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.350879 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.351047 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.353211 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.354048 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:10.355602 25104 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:10.350947 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.357342 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.358348 25031 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:10.219725 30204 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.290338 29994 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:10.290791 30165 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.337709 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.358057 25101 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:10.365626 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.430248 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.266892 30201 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20240426 13:21:10.433933 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.367434 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.433596 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:10.455811 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.516299 25100 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:10.368018 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.517386 25100 raft_consensus.cc:603] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:10.518078 25099 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:10.294319 29429 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.519606 25022 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:10.317862 30148 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:10.340121 30193 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:10.522354 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:10.561347 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 222257us
W20240426 13:21:10.560829 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 193450us
I20240426 13:21:10.566859 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.251001 29386 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:10.429811 30252 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20240426 13:21:10.571493 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.357319 30178 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:10.264921 29790 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.573710 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.366148 29775 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:10.350522 30210 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:10.354858 30199 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:10.572499 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:10.430228 30167 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:10.577919 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.578727 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.436839 29058 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:10.580778 25096 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:21:10.581472 25617 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:10.674293 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101858us
I20240426 13:21:10.580415 30179 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:10.484186 30049 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:10.434137 30011 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:10.579828 30193 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:10.588877 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:10.583889 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:10.589428 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:10.681985 25623 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:10.593679 25022 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:10.594848 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.609346 25099 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:10.590793 30178 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:10.684401 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.672066 25031 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:10.678459 24954 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:10.482885 29761 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
I20240426 13:21:10.524127 29909 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.581053 29386 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.687058 30255 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:10.700997 25617 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:10.684204 29446 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:10.687587 30219 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20240426 13:21:10.576764 30028 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
I20240426 13:21:10.597685 30202 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.248s
I20240426 13:21:10.709715 30202 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:21:10.710078 30202 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:10.710438 30185 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.619s	user 0.008s	sys 0.000s
W20240426 13:21:10.710788 30261 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
I20240426 13:21:10.710906 30185 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:10.712301 30226 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.918s	user 0.011s	sys 0.000s
I20240426 13:21:10.692422 30210 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.712163 30148 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:10.712785 30264 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20240426 13:21:10.713359 30148 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:10.713732 30262 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20240426 13:21:10.691991 29775 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.715968 29386 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:10.716238 29775 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.681345 29688 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:10.708891 30237 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:10.716609 30193 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.688129 30178 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:10.710932 30202 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.719857 30187 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:10.720374 30202 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:10.689723 30011 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.723035 30011 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.724260 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.724364 29630 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.724915 25105 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:10.572402 30152 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.729136 30255 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.730876 30185 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:10.731415 29790 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:10.732430 30257 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240426 13:21:10.732548 30204 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:10.716570 30253 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20240426 13:21:10.733953 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:10.734126 30253 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:10.735020 30253 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:10.735073 25022 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:10.678934 30199 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:10.737351 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:10.738267 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 172962us
I20240426 13:21:10.752489 25030 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:21:10.745208 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 678) took 1548 ms. Trace:
I20240426 13:21:10.757421 25105 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:10.758750 25105 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:10.761396 25103 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:10.761282 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:10.762588 25101 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:10.762437 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:10.754469 25029 rpcz_store.cc:276] 0426 13:21:09.196838 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.459758 (+262920us) service_pool.cc:225] Handling call
0426 13:21:09.484815 (+ 25057us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:09.516456 (+ 31641us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:09.776638 (+260182us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b18004b0220. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:21:09.954466 (+177828us) log.cc:842] Serialized 14 byte log entry
0426 13:21:10.212873 (+258407us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:10.213267 (+   394us) log.cc:842] Serialized 26 byte log entry
0426 13:21:10.213398 (+   131us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:10.213570 (+   172us) log.cc:842] Serialized 14 byte log entry
0426 13:21:10.213712 (+   142us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:10.213772 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:10.744968 (+531196us) raft_consensus.cc:1671] finished
0426 13:21:10.744996 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:10.745144 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92525,"prepare.queue_time_us":288218,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":83346,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":2474,"spinlock_wait_cycles":2919168,"thread_start_us":379784,"threads_started":2,"wal-append.queue_time_us":319988}
I20240426 13:21:10.790719 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:10.620234 29994 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.795588 25099 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20240426 13:21:10.735889 30253 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.801762 30253 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:10.804598 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:10.806201 30199 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.809726 30255 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.811792 29994 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:10.814685 29429 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:10.826819 25027 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:10.826990 30271 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:21:10.827567 30271 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:10.761868 30189 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:10.832875 25103 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20240426 13:21:10.791687 30217 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:10.736614 29446 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.796968 29790 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:10.843528 25099 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:10.849586 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.847590 25024 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:10.852077 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.853286 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.854286 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.855741 30167 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:10.856176 30179 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:10.856525 30201 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:10.857580 29446 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:10.855501 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.858392 30201 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:10.859426 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.860646 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:10.861418 30201 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:10.862540 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:10.863050 25623 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:10.863304 30179 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:10.864133 30179 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:10.863432 30185 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:10.864490 30179 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:10.865689 30256 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.682s	user 0.003s	sys 0.003s
I20240426 13:21:10.867288 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:10.828590 30271 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.868620 30271 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:10.869258 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 686) took 1083 ms. Trace:
I20240426 13:21:10.869591 25026 rpcz_store.cc:276] 0426 13:21:09.785649 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.833321 (+ 47672us) service_pool.cc:225] Handling call
0426 13:21:09.834136 (+   815us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:09.845764 (+ 11628us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:09.845805 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:09.846402 (+   597us) log.cc:842] Serialized 52 byte log entry
0426 13:21:10.014097 (+167695us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:10.070842 (+ 56745us) log.cc:842] Serialized 14 byte log entry
0426 13:21:10.204909 (+134067us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:10.204977 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:10.865196 (+660219us) spinlock_profiling.cc:243] Waited 6.91 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:10.868995 (+  3799us) raft_consensus.cc:1671] finished
0426 13:21:10.869084 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:10.869210 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164604,"prepare.queue_time_us":184415,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":134967,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":3779,"spinlock_wait_cycles":15890688,"thread_start_us":129543,"threads_started":1,"wal-append.queue_time_us":222068}
I20240426 13:21:10.871774 25026 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
W20240426 13:21:10.876331 30263 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.372s	user 0.000s	sys 0.001s
I20240426 13:21:10.878275 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:10.884209 25026 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:10.892239 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.892992 25026 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:10.898432 30148 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:10.853529 29790 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:10.901537 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:10.918310 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.919080 25026 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:10.920105 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:10.920789 25026 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:21:10.921511 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 241871us
I20240426 13:21:10.923244 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:10.924419 25623 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20240426 13:21:10.925781 25097 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
W20240426 13:21:10.925876 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46427.3us
I20240426 13:21:10.926173 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:10.878852 30253 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:10.924635 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.927851 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 897) took 1003 ms. Trace:
I20240426 13:21:10.930621 25097 rpcz_store.cc:276] 0426 13:21:09.924013 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.884601 (+960588us) service_pool.cc:225] Handling call
0426 13:21:10.925547 (+ 40946us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:10.927554 (+  2007us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:10.927790 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:10.929095 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:10.839794 30189 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.926674 29630 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:21:10.898802 30188 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20240426 13:21:10.880822 30210 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:10.957837 25096 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:10.924626 29790 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:10.985749 24956 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:10.985682 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.986196 24956 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:10.986932 24956 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:10.958030 25097 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:10.951946 29775 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:10.961194 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:10.903429 30179 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:10.964655 30253 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.013662 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.933606 30232 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20240426 13:21:10.975605 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:10.979127 24949 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 10. Current term is 11. Ops: []
I20240426 13:21:10.982373 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:11.015877 25617 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:11.016973 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.017163 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:10.964268 30237 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:10.989418 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.013613 25623 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20240426 13:21:10.956828 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:11.017724 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:11.024358 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.025617 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.025712 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.027056 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:11.028473 25622 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:11.036712 25101 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:10.984126 29630 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.042529 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.044458 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:11.044587 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:11.045116 30226 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:10.988600 29775 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:11.048075 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:11.052891 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.019733 30237 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:11.041457 30164 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.773s	user 0.000s	sys 0.009s
I20240426 13:21:11.056200 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.060683 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:11.061125 25099 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:11.061018 25096 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:11.062192 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:11.051915 25105 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:11.067374 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:11.036697 30202 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.068239 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:11.042771 29688 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.036798 30274 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:11.067543 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:11.056869 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:11.059405 30270 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20240426 13:21:11.056250 29775 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:11.100421 25026 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:11.067255 30237 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:11.060437 29956 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:21:11.075662 25617 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:11.165428 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 108618us
I20240426 13:21:11.102279 30251 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.098709 30164 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:11.100258 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 355343us
I20240426 13:21:11.102811 30210 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.171232 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2029) took 1144 ms. Trace:
I20240426 13:21:11.169633 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:11.067514 30197 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.097s
I20240426 13:21:11.132500 30277 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.179553 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:11.153309 29994 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:11.180459 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2676) took 1214 ms (client timeout 1595 ms). Trace:
I20240426 13:21:11.098979 29934 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.070s
I20240426 13:21:11.067667 30049 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.097s
I20240426 13:21:11.204205 30049 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:11.204545 30049 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.205096 30279 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.205255 30049 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.205577 30279 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.206386 30049 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:11.207406 30279 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.208570 30022 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:11.208966 30022 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:11.209719 30022 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.211136 30022 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:11.071743 30192 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20240426 13:21:11.212764 30192 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:11.213101 30192 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.213886 30192 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.215160 30192 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:11.221020 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2670) took 1947 ms (client timeout 1888 ms). Trace:
W20240426 13:21:11.221453 24950 rpcz_store.cc:269] 0426 13:21:09.273347 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.275122 (+  1775us) service_pool.cc:225] Handling call
0426 13:21:11.220855 (+1945733us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400078ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:11.220954 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2068548992}
I20240426 13:21:11.223779 30198 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:11.224179 30198 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.224435 29630 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.224999 30198 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.226053 30198 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:11.170225 29956 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:11.228929 30266 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.521s	user 0.002s	sys 0.000s
I20240426 13:21:11.168557 30201 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:11.131728 30202 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.170430 30188 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:11.177269 25031 rpcz_store.cc:276] 0426 13:21:10.026451 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.693641 (+667190us) service_pool.cc:225] Handling call
0426 13:21:10.703436 (+  9795us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:10.703618 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:10.765874 (+ 62256us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.170924 (+405050us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:11.170975 (+    51us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:11.171016 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.171073 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.171164 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7535,"prepare.queue_time_us":82503,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":36,"thread_start_us":397161,"threads_started":1}
I20240426 13:21:11.228402 29956 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:11.193398 24952 rpcz_store.cc:269] 0426 13:21:09.958059 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.969959 (+ 11900us) service_pool.cc:225] Handling call
0426 13:21:11.172387 (+1202428us) spinlock_profiling.cc:243] Waited 90.5 ms on lock 0x7b54003679d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:11.172558 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208162688}
I20240426 13:21:11.239464 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:11.132557 29688 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.190374 30278 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:11.199165 25026 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:11.269621 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:11.257108 30279 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.178072 30185 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20240426 13:21:11.176591 30251 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.154795 30253 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.132472 30274 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:11.270242 30048 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.013s	sys 0.056s
I20240426 13:21:11.303771 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2027) took 1278 ms. Trace:
I20240426 13:21:11.257968 30278 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.189535 30210 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.273527 30189 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.325634 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:11.183782 30164 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.235564 29994 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.321097 25023 rpcz_store.cc:276] 0426 13:21:10.025145 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.589667 (+564522us) service_pool.cc:225] Handling call
0426 13:21:10.589937 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:10.590167 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:10.590475 (+   308us) log.cc:842] Serialized 14 byte log entry
0426 13:21:10.924268 (+333793us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.303417 (+379149us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:11.303471 (+    54us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:11.303528 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.303588 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.303711 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1076,"prepare.queue_time_us":8047,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":121463,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":17982,"spinlock_wait_cycles":3867264,"thread_start_us":676597,"threads_started":2,"wal-append.queue_time_us":585627}
W20240426 13:21:11.320775 30048 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.013s	sys 0.056s
I20240426 13:21:11.280213 30281 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:11.303537 30275 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:11.323504 30179 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.232712 30277 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.398855 30277 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.400116 30281 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:11.401129 30281 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.313995 30279 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.409560 30279 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.410889 25031 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:11.412999 30210 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:11.414700 30265 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.706s	user 0.001s	sys 0.002s
I20240426 13:21:11.416302 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 672) took 2231 ms. Trace:
I20240426 13:21:11.416653 25024 rpcz_store.cc:276] 0426 13:21:09.184785 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.446915 (+262130us) service_pool.cc:225] Handling call
0426 13:21:09.447233 (+   318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:10.847389 (+1400156us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b540006d4a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:21:10.850486 (+  3097us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.416062 (+565576us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:11.416237 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1826790528}
I20240426 13:21:11.417587 30237 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.383633 30273 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.384951 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2397) took 1054 ms. Trace:
I20240426 13:21:11.386349 30202 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.331553 30278 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:11.392495 30280 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20240426 13:21:11.418689 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2022) took 1399 ms. Trace:
I20240426 13:21:11.418673 30273 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.418643 25102 rpcz_store.cc:276] 0426 13:21:10.330529 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.351999 (+ 21470us) service_pool.cc:225] Handling call
0426 13:21:10.433933 (+ 81934us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:10.792316 (+358383us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.792357 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:10.792869 (+   512us) log.cc:842] Serialized 26 byte log entry
0426 13:21:11.166279 (+373410us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:11.166325 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.166407 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:11.384741 (+218334us) raft_consensus.cc:1671] finished
0426 13:21:11.384774 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.384896 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58253,"raft.queue_time_us":85128,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":50857,"thread_start_us":664725,"threads_started":2,"wal-append.queue_time_us":584552}
I20240426 13:21:11.422016 30273 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.393466 29688 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.376638 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:11.425372 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 370595us
I20240426 13:21:11.395534 30189 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.419274 25028 rpcz_store.cc:276] 0426 13:21:10.019364 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.027544 (+  8180us) service_pool.cc:225] Handling call
0426 13:21:10.027932 (+   388us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:10.201960 (+174028us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.201990 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:10.202440 (+   450us) log.cc:842] Serialized 26 byte log entry
0426 13:21:10.355875 (+153435us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:10.355910 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:10.355965 (+    55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:11.414028 (+1058063us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:11.418417 (+  4389us) raft_consensus.cc:1671] finished
0426 13:21:11.418436 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.418635 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":173624,"raft.queue_time_us":175515,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":70022400,"thread_start_us":153075,"threads_started":1,"wal-append.queue_time_us":505883}
I20240426 13:21:11.418097 25024 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20240426 13:21:11.439517 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.443188 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2033) took 1103 ms. Trace:
I20240426 13:21:11.443538 25024 rpcz_store.cc:276] 0426 13:21:10.339364 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:11.417480 (+1078116us) service_pool.cc:225] Handling call
0426 13:21:11.417771 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:11.441329 (+ 23558us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.443140 (+  1811us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:11.442005 30190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.256s	user 0.011s	sys 0.009s
W20240426 13:21:11.450314 30190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.267s	user 0.011s	sys 0.010s
I20240426 13:21:11.454595 24951 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20240426 13:21:11.457917 30237 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.458406 30164 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:11.459692 30269 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.748s	user 0.000s	sys 0.002s
I20240426 13:21:11.461544 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2776) took 1402 ms. Trace:
I20240426 13:21:11.462041 25022 rpcz_store.cc:276] 0426 13:21:10.059137 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.735906 (+676769us) service_pool.cc:225] Handling call
0426 13:21:10.736254 (+   348us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:10.736449 (+   195us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.736478 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:10.747969 (+ 11491us) log.cc:842] Serialized 26 byte log entry
0426 13:21:11.234705 (+486736us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:11.271084 (+ 36379us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.271307 (+   223us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.271408 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:11.461338 (+189930us) raft_consensus.cc:1671] finished
0426 13:21:11.461362 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.461490 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25722,"prepare.queue_time_us":36032,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2363,"raft.queue_time_us":122641,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":407493,"threads_started":1,"wal-append.queue_time_us":662811}
I20240426 13:21:11.464279 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2780) took 1400 ms. Trace:
I20240426 13:21:11.464471 25022 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:11.464671 25030 rpcz_store.cc:276] 0426 13:21:10.063853 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.764467 (+700614us) service_pool.cc:225] Handling call
0426 13:21:10.798091 (+ 33624us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:10.798495 (+   404us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.798538 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:10.799175 (+   637us) log.cc:842] Serialized 52 byte log entry
0426 13:21:11.055486 (+256311us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:11.163177 (+107691us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.163582 (+   405us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.164204 (+   622us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.164269 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:11.204613 (+ 40344us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b18004993c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:11.464074 (+259461us) raft_consensus.cc:1671] finished
0426 13:21:11.464098 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.464219 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12861,"prepare.queue_time_us":138936,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":5323,"raft.queue_time_us":58656,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":7898,"spinlock_wait_cycles":12549888,"thread_start_us":203816,"threads_started":1,"wal-append.queue_time_us":521446}
I20240426 13:21:11.456219 30036 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.466997 25030 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:11.466804 30036 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.442034 30049 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
I20240426 13:21:11.468688 30036 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.475139 30049 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:21:11.476302 30274 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:11.477973 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.012s	sys 0.020s
W20240426 13:21:11.479009 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.012s	sys 0.020s
I20240426 13:21:11.481791 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.448168 30232 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:21:11.483842 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.484604 25099 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:11.484880 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.485751 25100 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:11.446048 30189 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.509011 25100 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:11.503959 25031 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 13:21:11.454571 30202 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.487568 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:11.514145 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:11.516413 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.515796 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.520669 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.521005 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:11.522186 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.523377 25102 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:11.528440 25589 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.006s	sys 0.014s
W20240426 13:21:11.528788 25589 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.006s	sys 0.014s
W20240426 13:21:11.529145 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 602464us
W20240426 13:21:11.551817 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 280467us
W20240426 13:21:11.567559 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45461.9us
I20240426 13:21:11.568625 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:11.570557 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:11.571115 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.571614 25096 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:11.527668 30021 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:11.527539 30190 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:11.520323 30163 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.583395 25620 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:21:11.584745 25620 raft_consensus.cc:595] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:11.585196 25620 raft_consensus.cc:603] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:11.582417 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.586416 25617 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:11.601316 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.603981 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.531646 30125 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:11.614503 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:11.620458 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.622028 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.622602 24953 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:11.622613 24948 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:21:11.524444 30202 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.569967 30220 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:11.575285 30021 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:11.635986 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.632287 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.579483 29688 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:11.639137 29688 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.639601 29688 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:11.640178 30241 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.134s
W20240426 13:21:11.641597 30298 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20240426 13:21:11.642570 30290 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20240426 13:21:11.574411 30273 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:11.583566 30210 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:21:11.644486 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2787) took 1446 ms. Trace:
I20240426 13:21:11.645740 30202 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.644901 25027 rpcz_store.cc:276] 0426 13:21:10.197961 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.827607 (+629646us) service_pool.cc:225] Handling call
0426 13:21:10.827949 (+   342us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:10.828397 (+   448us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.828428 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:10.828869 (+   441us) log.cc:842] Serialized 26 byte log entry
0426 13:21:11.164764 (+335895us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:11.262506 (+ 97742us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.262727 (+   221us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.262816 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:11.644274 (+381458us) raft_consensus.cc:1671] finished
0426 13:21:11.644298 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:11.644428 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37659,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":73180,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":11894,"thread_start_us":335497,"threads_started":1,"wal-append.queue_time_us":620554}
I20240426 13:21:11.615677 30163 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.653941 30163 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:11.659379 30178 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.941s	user 0.004s	sys 0.005s
I20240426 13:21:11.661487 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:11.662978 30198 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:11.631429 29790 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.001s
I20240426 13:21:11.663641 30198 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.665745 30232 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:11.665993 30198 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.666242 30232 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.666687 30198 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:11.667290 30198 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.612996 30190 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:11.668987 30164 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.669551 30164 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.627518 30073 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:11.673164 30148 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:11.672881 30073 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.673658 30148 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:11.675155 30073 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.674714 30148 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.613169 29761 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:11.611184 30197 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:11.677704 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49553.4us
I20240426 13:21:11.630357 25069 leader_election.cc:398] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:11.638101 30220 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.679566 24950 raft_consensus.cc:2396] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20240426 13:21:11.680755 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50507.7us
I20240426 13:21:11.671821 30164 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.677304 30223 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:11.710021 30164 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.710098 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.659899 30178 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.732784 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.710968 30189 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.711979 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.664436 29790 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.714066 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2680) took 1754 ms. Trace:
I20240426 13:21:11.677873 30021 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:11.748085 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.748713 24950 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:11.749660 30182 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:11.750267 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:11.750365 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.751250 30210 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:11.751665 30210 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:11.751955 30210 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:11.753026 30049 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:11.753434 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.753501 30049 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.754817 30278 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.755456 30049 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.709069 30274 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20240426 13:21:11.745644 24954 rpcz_store.cc:276] 0426 13:21:09.959327 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.960118 (+   791us) service_pool.cc:225] Handling call
0426 13:21:09.972061 (+ 11943us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:10.690989 (+718928us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.713848 (+1022859us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540004b854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:11.714013 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1624182272}
I20240426 13:21:11.756760 30274 consensus_queue.cc:1225] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240426 13:21:11.757278 25028 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:11.715723 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2677) took 1757 ms (client timeout 1652 ms). Trace:
I20240426 13:21:11.715889 30223 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:11.758368 24947 rpcz_store.cc:269] 0426 13:21:09.958453 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.959928 (+  1475us) service_pool.cc:225] Handling call
0426 13:21:11.715503 (+1755575us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540004b354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:11.715645 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2504743808}
I20240426 13:21:11.758865 30251 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:11.720361 25069 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.652s (SENT) [suppressed 3 similar messages]
I20240426 13:21:11.759591 25103 raft_consensus.cc:595] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:11.760157 30178 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.759955 25103 raft_consensus.cc:603] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:11.761197 30178 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:11.761600 25097 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:11.717713 24952 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:11.762102 30036 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:11.743681 30164 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.762454 30036 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:11.727167 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.727365 30198 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.728255 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.763276 30036 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.763823 30164 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.730020 30220 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.636730 30296 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
I20240426 13:21:11.677242 30197 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.643676 30273 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.619910 30125 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.684687 30232 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.708314 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.624364 30199 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
I20240426 13:21:11.767601 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:11.799434 24955 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:11.765249 24953 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:11.607591 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:11.808919 30166 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:11.809034 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 201525us
I20240426 13:21:11.809329 30166 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:11.809733 30166 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:11.809827 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:11.807240 25623 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:11.764791 30036 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:11.828310 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:11.840000 30226 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.007s	sys 0.022s
W20240426 13:21:11.800048 29630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.008s	sys 0.028s
W20240426 13:21:11.856252 29630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.009s	sys 0.028s
W20240426 13:21:11.856384 30226 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.007s	sys 0.022s
I20240426 13:21:11.847723 30185 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:11.791268 30021 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:11.860639 29909 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.147s	user 0.006s	sys 0.025s
W20240426 13:21:11.877538 30193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.143s	user 0.006s	sys 0.022s
I20240426 13:21:11.854660 30271 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:11.889202 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.017s	sys 0.029s
W20240426 13:21:11.890007 30254 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.705s	user 0.002s	sys 0.008s
I20240426 13:21:11.892344 30278 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.892666 30273 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.758865 30223 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.893693 29790 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.895515 30210 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.896250 30197 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:11.895570 30166 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:11.904755 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.018s	sys 0.029s
I20240426 13:21:11.909917 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:11.902904 29909 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.193s	user 0.006s	sys 0.026s
I20240426 13:21:11.955255 29994 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.956367 30125 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.960083 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:11.904744 30152 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.174s	user 0.015s	sys 0.025s
I20240426 13:21:11.913409 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:11.964217 30021 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.964375 25622 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:11.966419 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2034) took 1621 ms. Trace:
I20240426 13:21:11.966792 29688 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.966876 25028 rpcz_store.cc:276] 0426 13:21:10.345304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:11.441005 (+1095701us) service_pool.cc:225] Handling call
0426 13:21:11.757112 (+316107us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:11.757674 (+   562us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:11.966240 (+208566us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b5400056ed0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:11.966364 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":697365376}
W20240426 13:21:11.904103 30193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.185s	user 0.006s	sys 0.022s
I20240426 13:21:11.970007 30217 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:11.970425 30217 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:11.972879 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.974272 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:11.974907 24952 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:11.977922 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2189) took 1001 ms. Trace:
I20240426 13:21:11.978278 24955 rpcz_store.cc:276] 0426 13:21:10.976447 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.976745 (+   298us) service_pool.cc:225] Handling call
0426 13:21:11.977717 (+1000972us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:11.977843 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1795411712}
I20240426 13:21:11.968879 29688 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.908532 30271 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:11.983557 30271 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:11.939005 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:11.940146 29775 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.776s	user 0.010s	sys 0.000s
I20240426 13:21:11.941711 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.985198 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.985651 25618 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:11.986044 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.943987 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.944964 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.945675 30210 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:11.921629 30223 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:11.946730 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:11.946863 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:11.906081 30192 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.008s
I20240426 13:21:11.990258 25615 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:11.992424 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45627.6us
I20240426 13:21:11.999778 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:11.971133 30217 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:11.986785 29775 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.004951 24948 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:12.005821 25615 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:12.009049 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.009779 24952 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:12.009985 24956 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.012640 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:12.003638 30152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.273s	user 0.015s	sys 0.026s
I20240426 13:21:12.015945 25622 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:12.020234 25615 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:21:12.002768 30197 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:12.021056 24950 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:12.006697 30271 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:12.019011 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.034029 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2031) took 1724 ms. Trace:
W20240426 13:21:12.037334 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 220603us
I20240426 13:21:12.029832 24951 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:12.050632 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.057392 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.058810 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:12.051414 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:12.059185 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.061167 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.032913 30165 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.072646 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2785) took 1892 ms. Trace:
I20240426 13:21:12.024386 30197 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:12.073114 25029 rpcz_store.cc:276] 0426 13:21:10.179959 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.788308 (+608349us) service_pool.cc:225] Handling call
0426 13:21:10.830558 (+ 42250us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:10.850367 (+ 19809us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:10.850399 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:10.885228 (+ 34829us) log.cc:842] Serialized 52 byte log entry
0426 13:21:11.330724 (+445496us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:11.382631 (+ 51907us) log.cc:842] Serialized 14 byte log entry
0426 13:21:11.861108 (+478477us) log.cc:842] Serialized 14 byte log entry
0426 13:21:12.072235 (+211127us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:12.072333 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:12.072400 (+    67us) raft_consensus.cc:1671] finished
0426 13:21:12.072418 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:12.072583 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":229225,"prepare.queue_time_us":51827,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":4060,"raft.queue_time_us":32543,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":5745,"thread_start_us":850514,"threads_started":2,"wal-append.queue_time_us":533665}
I20240426 13:21:12.038801 30274 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:12.036397 30271 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:12.035856 30217 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.060127 25025 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
W20240426 13:21:12.029592 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 219836us
I20240426 13:21:12.078222 24956 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20240426 13:21:12.104709 24956 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:21:12.104991 24956 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20240426 13:21:12.105257 24956 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:12.106603 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.107717 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.113034 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.114243 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.114766 25105 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:12.114956 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.115541 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.105991 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2188) took 1138 ms. Trace:
I20240426 13:21:12.116176 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:12.117298 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.117465 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.118229 25099 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:12.071861 30296 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.119346 25099 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:12.116175 25104 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:12.120648 25099 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:21:12.103098 29759 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.121388 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.116729 24956 rpcz_store.cc:276] 0426 13:21:10.967168 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:10.967408 (+   240us) service_pool.cc:225] Handling call
0426 13:21:10.985647 (+ 18239us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:12.009547 (+1023900us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b58005cc0b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:21:12.105710 (+ 96163us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:12.105931 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2089708416}
I20240426 13:21:12.124682 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.131273 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:12.132855 25104 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:21:12.123772 29759 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.138553 30294 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.143253 30163 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.143771 30163 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.145715 30163 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:12.146415 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 89136.4us
I20240426 13:21:12.148154 25620 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20240426 13:21:12.150569 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.151242 29688 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:12.085101 29790 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20240426 13:21:12.151882 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.118258 30274 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:12.153312 30274 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.154520 30274 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:12.154547 30226 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:12.155037 30210 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.157100 30165 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.150879 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.159263 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.160151 24956 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.088199 30191 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:12.161648 24953 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:12.165391 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:12.160015 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2795) took 1198 ms. Trace:
I20240426 13:21:12.162076 30301 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.005s
I20240426 13:21:12.100596 29956 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:12.085278 30232 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.089270 30197 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.100309 30271 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.166224 25022 rpcz_store.cc:276] 0426 13:21:10.961779 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:11.465175 (+503396us) service_pool.cc:225] Handling call
0426 13:21:11.465433 (+   258us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:11.465739 (+   306us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:21:11.465995 (+   256us) log.cc:842] Serialized 14 byte log entry
0426 13:21:12.121927 (+655932us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:12.136481 (+ 14554us) log.cc:842] Serialized 26 byte log entry
0426 13:21:12.136648 (+   167us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:12.136710 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:12.136826 (+   116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:12.159770 (+ 22944us) raft_consensus.cc:1671] finished
0426 13:21:12.159803 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:12.159950 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":213179,"prepare.queue_time_us":283,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":231022,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":442379,"threads_started":1,"wal-append.queue_time_us":604749}
I20240426 13:21:12.155570 30210 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:12.121503 30213 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20240426 13:21:12.169411 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:12.153252 30316 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:12.088613 30315 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:12.118605 30201 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20240426 13:21:12.164463 30241 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.010s	sys 0.017s
W20240426 13:21:12.177661 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55400.6us
I20240426 13:21:12.180846 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:12.224670 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 43915.3us
I20240426 13:21:12.181531 30163 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:12.182564 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76059.4us
I20240426 13:21:12.250864 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:12.192186 30073 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.012s	sys 0.013s
W20240426 13:21:12.252704 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 393673us
I20240426 13:21:12.193898 24956 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:12.125376 30296 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.174711 30237 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:12.206744 24850 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:12.207808 30204 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.209028 29759 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.203701 30255 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:12.218806 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:12.175258 30191 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:12.220314 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 685) took 2490 ms (client timeout 1598 ms). Trace:
I20240426 13:21:12.220593 29775 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.192231 30226 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
W20240426 13:21:12.178074 30281 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.019s	sys 0.014s
I20240426 13:21:12.222301 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2021) took 1259 ms. Trace:
I20240426 13:21:12.233503 30166 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.178267 29956 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:12.235571 30210 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.222091 30179 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.243199 30271 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.252754 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.280289 24850 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 6 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 11 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:12.282150 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.282647 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.283267 25027 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.124915 30312 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.284346 30312 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.284612 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.285534 25030 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.286183 30312 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.286271 30296 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.286747 30312 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.286813 30296 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:12.287019 30163 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:12.264317 30255 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:12.287468 30312 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:12.256073 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.291152 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.034520 25026 rpcz_store.cc:276] 0426 13:21:10.309648 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:11.260670 (+951022us) service_pool.cc:225] Handling call
0426 13:21:11.261057 (+   387us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:11.261317 (+   260us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:11.320494 (+ 59177us) log.cc:842] Serialized 14 byte log entry
0426 13:21:12.033437 (+712943us) log.cc:842] Serialized 14 byte log entry
0426 13:21:12.033642 (+   205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:12.033696 (+    54us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:12.033752 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:12.033839 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:12.033967 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":234666,"prepare.queue_time_us":59538,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":1254,"thread_start_us":490735,"threads_started":1,"wal-append.queue_time_us":674248}
I20240426 13:21:12.292356 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.293033 30210 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.273772 30201 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.293706 30201 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.293913 30191 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.295318 30201 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:12.296023 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 177) took 19282 ms (client timeout 15000 ms). Trace:
I20240426 13:21:12.291945 30294 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.264453 30204 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.292560 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.298209 30294 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.298602 30204 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:12.283735 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.266809 30198 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20240426 13:21:12.249289 30241 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.011s	sys 0.017s
W20240426 13:21:12.253803 30073 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.012s	sys 0.013s
I20240426 13:21:12.305917 30164 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240426 13:21:12.263407 25025 rpcz_store.cc:269] 0426 13:21:09.730235 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:09.746907 (+ 16672us) service_pool.cc:225] Handling call
0426 13:21:12.220130 (+2473223us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:12.220245 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5138289408}
W20240426 13:21:12.263984 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45290.9us
I20240426 13:21:12.266464 25097 rpcz_store.cc:276] 0426 13:21:10.963221 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:11.025409 (+ 62188us) service_pool.cc:225] Handling call
0426 13:21:12.222057 (+1196648us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:12.222233 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1690721536}
I20240426 13:21:12.267848 25028 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:12.267995 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.014s	sys 0.036s
I20240426 13:21:12.252869 30297 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:12.201351 30178 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20240426 13:21:12.272543 25029 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20240426 13:21:12.301167 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.270689 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.258827 29688 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:12.259254 30237 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:12.260077 30190 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:12.275521 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.277072 29956 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.277879 30179 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.275477 30152 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.287904 30255 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:12.287691 30021 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:12.330472 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.330687 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.330477 30021 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:12.318919 30279 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:12.257122 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:12.332087 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.332422 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:12.332734 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 75735.3us
I20240426 13:21:12.179769 30165 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:12.307902 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.014s	sys 0.037s
W20240426 13:21:12.336548 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 709723us
I20240426 13:21:12.266451 30193 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.261029 30187 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:21:12.311640 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:12.317057 30237 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.312441 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.314680 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.315714 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.317906 30314 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.313711 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:12.317574 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46955.1us
I20240426 13:21:12.321997 30178 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
W20240426 13:21:12.296408 24852 rpcz_store.cc:269] 0426 13:20:53.013675 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:53.038402 (+ 24727us) service_pool.cc:225] Handling call
0426 13:21:12.295944 (+19257542us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:07.823329 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:07.824142 (+   813us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:07.824198 (+    56us) write_op.cc:427] Acquired schema lock
0426 13:21:07.824215 (+    17us) tablet.cc:660] Decoding operations
0426 13:21:07.829687 (+  5472us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:07.829770 (+    83us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:07.829810 (+    40us) tablet.cc:683] Acquiring locks for 9 operations
0426 13:21:07.830674 (+   864us) tablet.cc:699] Row locks acquired
0426 13:21:07.830718 (+    44us) write_op.cc:260] PREPARE: finished
0426 13:21:07.839773 (+  9055us) write_op.cc:270] Start()
0426 13:21:07.853546 (+ 13773us) write_op.cc:275] Timestamp: P: 1714137667830835 usec, L: 0
0426 13:21:07.862985 (+  9439us) op_driver.cc:331] REPLICATION: starting
0426 13:21:07.986214 (+123229us) log.cc:842] Serialized 3547 byte log entry
0426 13:21:08.962261 (+976047us) op_driver.cc:444] REPLICATION: finished
0426 13:21:09.138777 (+176516us) write_op.cc:300] APPLY: starting
0426 13:21:09.138887 (+   110us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:09.201782 (+ 62895us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:09.201868 (+    86us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:09.217633 (+ 15765us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:09.217971 (+   338us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0426 13:21:09.218013 (+    42us) write_op.cc:311] APPLY: finished
0426 13:21:09.218361 (+   348us) log.cc:842] Serialized 89 byte log entry
0426 13:21:09.219074 (+   713us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:09.219203 (+   129us) write_op.cc:446] Released schema lock
0426 13:21:09.219287 (+    84us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:09.219586 (+   299us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:09.219614 (+    28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":10433751,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":166347,"apply.run_cpu_time_us":13109,"apply.run_wall_time_us":81165,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41765,"num_ops":9,"prepare.queue_time_us":149994,"prepare.run_cpu_time_us":37045,"prepare.run_wall_time_us":507219,"raft.queue_time_us":380613,"raft.run_cpu_time_us":74794,"raft.run_wall_time_us":176727,"replication_time_us":1093334,"thread_start_us":535300,"threads_started":4,"wal-append.queue_time_us":418443,"wal-append.run_cpu_time_us":67574,"wal-append.run_wall_time_us":1757987}]]}
I20240426 13:21:12.303123 30198 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.322587 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.310920 30317 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:12.327908 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.355113 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:12.307649 29759 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:12.330320 30152 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.333331 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:12.271293 30281 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.019s	sys 0.014s
I20240426 13:21:12.309163 30297 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.311982 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.339740 30204 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:12.363936 25027 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:12.310909 30312 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.339099 25030 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:12.370380 30312 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.371340 30179 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.342927 29775 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:12.372305 29775 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:12.372638 29775 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:12.340581 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:12.373577 30312 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.287645 30163 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.376221 30163 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.376256 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:12.377331 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44057.1us
I20240426 13:21:12.377522 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.378348 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:12.378645 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.379146 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.380167 30188 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.380529 30274 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:12.381089 30274 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.381512 30274 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.381595 30036 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:12.342304 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.328246 30336 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.343029 24950 raft_consensus.cc:2396] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20240426 13:21:12.365175 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.343063 25619 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:12.357507 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.358611 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.352187 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.359366 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.362439 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.341842 30165 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.353236 25023 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:12.363929 30297 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.366418 29761 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.360855 30317 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.352523 30314 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.339130 30021 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.340803 30193 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.359362 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.380374 29630 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:12.317214 30190 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.317621 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.380725 30188 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.354318 30198 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.380170 30223 raft_consensus.cc:2741] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20240426 13:21:12.432910 29775 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.382441 29429 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:12.433202 30163 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.433957 29429 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:12.434739 29429 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.379233 25101 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:12.435961 30271 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.436193 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:12.436750 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.405491 25099 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.438001 30193 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.439561 30197 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.405647 25102 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.354141 30328 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20240426 13:21:12.440929 24955 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:12.442484 30197 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.384928 25105 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:12.410090 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.410735 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.411311 25615 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:21:12.446954 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.447188 30237 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.439643 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.422366 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.350318 30178 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:12.330190 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.449995 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.425693 29761 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.426015 30190 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.364316 30204 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.429543 30165 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.283293 24850 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 10, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:12.430780 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.431200 30314 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.436810 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.447701 25104 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:12.462030 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.463793 25025 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.396273 30152 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:12.424193 25022 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:12.434916 29775 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:12.458832 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95851.1us
I20240426 13:21:12.428761 25023 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:12.361645 30182 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.454877 25099 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:12.392552 30179 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.464881 24949 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20240426 13:21:12.466969 25615 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:12.467165 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.416548 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:12.447752 30210 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:12.470162 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:12.488284 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.486231 25100 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:12.455806 29759 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:12.494266 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.504289 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:12.461319 30036 consensus_queue.cc:1225] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:21:12.506332 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.457633 30313 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:12.461458 30204 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.524066 25100 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:12.464198 30021 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.521478 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.529909 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.531105 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.525920 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.532255 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.532800 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:12.504849 24850 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 12, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 12 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:12.533095 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.536563 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:12.540652 25104 raft_consensus.cc:595] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.541110 25104 raft_consensus.cc:603] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:12.536834 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.529829 30210 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:12.539132 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.539594 25098 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:12.481230 30193 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.543731 24955 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:12.544829 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:12.548573 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.549499 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.547096 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.543658 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.543439 30210 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:12.553767 30210 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.554685 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.555209 25022 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:12.556107 30182 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:12.557848 30226 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:12.558337 30226 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.506654 30152 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.560314 30152 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.561484 30340 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:12.561856 30340 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.562160 30340 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.562217 30152 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.562842 30148 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:12.563118 30317 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.563661 30317 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:12.563953 30317 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.564282 30317 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.525859 30314 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:12.569036 30314 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.570906 30314 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.572827 29761 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.541544 30189 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:12.573765 30189 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:12.574146 30189 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.576907 30188 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:12.544459 30163 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.582660 30198 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.584149 24953 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:12.525486 29909 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.593142 30255 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.595125 30255 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.596755 30237 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.597885 24850 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 4 to 6, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 6 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:12.602593 25029 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.606293 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.607322 24951 raft_consensus.cc:2396] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20240426 13:21:12.608094 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.627650 24947 raft_consensus.cc:2354] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:12.629079 25031 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.628731 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:12.639431 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.584038 29759 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.641594 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.645292 30313 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.645684 30313 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.506979 30179 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.627454 30193 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.628638 30226 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.648002 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.666085 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:12.666656 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:12.669741 24995 leader_election.cc:398] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:12.670058 24995 leader_election.cc:401] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20240426 13:21:12.653985 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.662740 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.662797 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.665446 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.654924 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.654495 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.677126 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:12.701673 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.671666 25030 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:12.664321 25097 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:12.703616 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:12.704923 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50056.4us
I20240426 13:21:12.705878 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.701673 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.712123 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.713830 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.714411 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.715260 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.715576 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.662235 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:12.718312 25030 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.719792 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.725646 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.745798 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.658828 30011 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:12.652876 30193 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.650950 30179 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.714262 30163 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:12.702808 30182 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:12.784301 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.784580 30182 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.785703 25025 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240426 13:21:12.786360 30182 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.657927 30190 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.788127 30190 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.774606 30011 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:12.789003 30195 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:12.789181 30011 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.752939 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.790220 30011 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.779562 29956 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:12.706939 30315 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.740909 30210 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.741485 29688 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.711828 30165 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20240426 13:21:12.792701 30165 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:12.793059 30165 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:12.793802 30165 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.660495 24850 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 8 to 9, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 9 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:12.794950 30165 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.714665 30314 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.796391 30314 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.796983 30210 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.790336 30195 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:12.798339 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.798857 30314 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.714504 30293 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:12.756685 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:12.741641 30188 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:12.800334 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:12.756795 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.763150 30028 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:12.790323 25030 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:12.766283 24949 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:21:12.768013 25022 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:12.657907 30154 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:12.702347 30294 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.725183 30187 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20240426 13:21:12.768911 30297 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.819262 30297 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.771036 30351 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:12.661072 30281 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.820695 30187 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:12.821221 30187 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:12.822180 30187 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.823629 30187 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:12.661772 30317 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.652345 30274 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.828163 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:12.772585 30350 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.129s	user 0.002s	sys 0.000s
W20240426 13:21:12.774107 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48549.4us
W20240426 13:21:12.775802 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60758.5us
I20240426 13:21:12.751791 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:12.711863 30296 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:12.756390 30344 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.112s	user 0.002s	sys 0.000s
I20240426 13:21:12.806329 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.743580 30304 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:12.816218 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.834520 24954 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.835232 24955 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:21:12.816501 25028 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:12.817157 30028 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:12.798856 30195 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:12.820493 30281 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.653542 30198 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.725030 29909 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.819909 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.779390 30036 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:12.830526 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.830660 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.853420 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.857370 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:12.830341 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 355124us
I20240426 13:21:12.835822 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.837872 25101 raft_consensus.cc:2396] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20240426 13:21:12.838335 30028 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:12.814440 30154 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:12.657474 30178 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.857944 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.843827 30204 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.838969 30304 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:12.860987 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.833040 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.830770 29688 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:21:12.837430 30191 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20240426 13:21:12.869400 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.855891 29775 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.884027 30154 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.893057 25104 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:12.858968 30314 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.860379 30253 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:12.880932 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47967.6us
I20240426 13:21:12.880823 30195 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.882053 30237 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:21:12.916518 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:12.877547 30255 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:12.882901 29429 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:12.883147 30028 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.885203 30184 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:12.886057 30193 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:12.886241 30274 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:12.886914 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:12.887411 30281 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.840377 30189 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.932332 30028 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:12.874575 30304 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.933109 25029 raft_consensus.cc:2354] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16.
I20240426 13:21:12.933365 25027 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:21:12.840135 24850 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 9 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 11 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:12.895824 24954 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:12.896293 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:12.857615 30317 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.902264 30297 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:12.904963 25028 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:12.906582 30164 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:12.940497 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:12.941107 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54255.7us
I20240426 13:21:12.908511 29688 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:12.919145 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:12.894121 30349 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.245s	user 0.000s	sys 0.002s
I20240426 13:21:12.953653 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.919651 30314 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:12.956543 30281 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:12.956920 30281 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:12.957233 30281 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.957971 30204 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.937388 30340 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.960714 30336 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.961115 30340 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:12.934049 30313 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:12.951843 30237 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.011s
I20240426 13:21:12.957255 30314 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.964332 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:12.975028 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:12.963686 30313 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:12.924259 25022 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:12.916515 30195 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:12.937266 29956 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.907485 30241 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:12.941238 25616 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.908807 30315 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:12.943135 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.944064 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:12.945813 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.946133 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.948640 24951 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Deduplicated request from leader. Original: 13.10->[14.11-14.11]   Dedup: 14.11->[]
I20240426 13:21:12.939116 30189 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:12.932857 30154 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:12.920751 30353 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:12.979583 25617 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 13.10->[14.11-14.11]   Dedup: 14.11->[]
I20240426 13:21:13.008361 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:12.936432 30217 raft_consensus.cc:2741] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20240426 13:21:12.950712 29688 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:12.982929 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.013497 25622 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:21:12.988276 24951 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:12.940977 30164 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:12.984055 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:13.019639 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:12.922308 25615 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:12.952868 30178 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:13.010285 24954 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20240426 13:21:13.008984 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:12.950986 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.978736 30226 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:12.952495 30301 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.062s
I20240426 13:21:12.950392 30274 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:13.026907 25028 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:13.022568 24953 raft_consensus.cc:2354] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:21:13.019282 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:13.026439 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 75527.2us
I20240426 13:21:13.007741 30201 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.011720 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:13.030725 25029 raft_consensus.cc:2354] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:21:13.008571 30073 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.030062 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.029016 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:12.974625 30188 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20240426 13:21:13.033444 25070 leader_election.cc:398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:21:13.039408 25070 leader_election.cc:401] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20240426 13:21:13.033265 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.040369 25022 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:12.961216 30336 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.043555 30336 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.023340 30148 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:13.034765 30296 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:13.037096 30340 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:13.029428 30226 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.046705 30226 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:13.047431 30182 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:13.047978 30182 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.048609 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.048837 30182 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.049460 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.050410 30021 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:13.050868 30021 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.052310 30296 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:13.052739 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:13.052850 30296 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.053594 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:13.054875 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:13.054919 30296 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.023881 30036 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:13.032097 30193 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:12.990736 29956 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.056529 30193 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.058210 29956 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:13.058393 30193 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:13.060333 30164 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.061002 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:13.061559 30164 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:13.061745 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:13.039716 30201 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.063779 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:13.048905 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:13.073359 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.073540 25029 pending_rounds.cc:76] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 12
I20240426 13:21:13.074278 25029 pending_rounds.cc:97] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20240426 13:21:13.074587 25029 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:13.069906 25026 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:13.076259 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 351944us
I20240426 13:21:13.077589 25616 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:13.077075 24955 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:13.073956 25027 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:13.082134 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.072438 24950 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:13.084725 24949 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:13.064379 30201 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.065675 24953 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:13.088980 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:13.085798 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.091638 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:13.106783 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.107270 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:13.108026 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:13.116338 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.076095 30340 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.122225 30188 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:13.122732 30188 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.123208 30188 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.124096 30255 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:13.124585 30255 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.124971 30255 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:13.074023 30148 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:13.128428 30148 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.130438 30148 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.063427 24850 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 11, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:13.131697 24850 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:13.134213 24850 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 4 to 5, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 5 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:13.134907 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.135547 25028 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:13.058454 30202 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:13.137375 25069 leader_election.cc:398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:21:13.143832 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.137928 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.145108 25027 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:13.064483 30184 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:13.086584 30321 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:21:13.087996 30351 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:13.127560 30281 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.075361 29956 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.146062 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.187304 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:13.107656 30336 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.237798 25026 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20240426 13:21:13.051687 30021 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.190260 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.115940 30189 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
W20240426 13:21:13.204520 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58521.9us
W20240426 13:21:13.204602 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51189.7us
I20240426 13:21:13.142050 30179 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:13.065127 30198 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20240426 13:21:13.143103 30148 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:13.066159 30073 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.220854 30237 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:13.224529 24850 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 3 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 5 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:13.228083 30281 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.122650 30340 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.240929 30204 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.211297 30314 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:13.152472 30351 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.188138 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:13.290267 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102221us
I20240426 13:21:13.211622 30191 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:13.209530 30353 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:13.251708 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.298864 25028 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.252146 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:13.300200 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48157.2us
I20240426 13:21:13.265982 24953 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20240426 13:21:13.109838 29688 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.293092 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:13.295329 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:13.308238 24949 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:13.295571 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:13.273555 30189 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.311908 29759 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:13.267987 30237 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:13.312882 30148 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:13.313273 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.005s	sys 0.027s
W20240426 13:21:13.324496 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.006s	sys 0.027s
W20240426 13:21:13.314060 30355 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20240426 13:21:13.316356 24954 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:13.312410 25022 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:13.211804 29956 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.304471 30351 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.321879 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.298405 30191 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.293166 30314 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:13.338624 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:13.269440 30202 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.057s
I20240426 13:21:13.243521 30336 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.359992 25028 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:13.361903 24947 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.364269 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:13.321732 25024 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.007s	sys 0.023s
W20240426 13:21:13.366962 25024 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.007s	sys 0.023s
I20240426 13:21:13.310058 30281 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:13.314733 30011 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:13.327364 30353 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:13.349994 25102 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20240426 13:21:13.349776 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.371814 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.375443 25098 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:13.341253 30187 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.302888 30204 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.377317 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.352378 30313 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:13.363165 30191 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:13.367853 30314 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:13.400482 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2215) took 1071 ms. Trace:
W20240426 13:21:13.385289 30354 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20240426 13:21:13.409596 24956 rpcz_store.cc:276] 0426 13:21:12.329236 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.329366 (+   130us) service_pool.cc:225] Handling call
0426 13:21:12.329693 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:12.333060 (+  3367us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:12.333089 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:12.564551 (+231462us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:12.564894 (+   343us) log.cc:842] Serialized 26 byte log entry
0426 13:21:12.855888 (+290994us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:12.855922 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:12.855979 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:13.384780 (+528801us) raft_consensus.cc:1671] finished
0426 13:21:13.384799 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:13.400404 (+ 15605us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3206,"raft.queue_time_us":3342,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":6037,"spinlock_wait_cycles":6138240,"thread_start_us":290744,"threads_started":1,"wal-append.queue_time_us":577497}
I20240426 13:21:13.373282 30281 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.418612 30336 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.395838 30011 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:13.368217 29429 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20240426 13:21:13.408330 30189 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.373728 30193 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:13.396345 30201 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:13.346850 30253 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
W20240426 13:21:13.400887 30357 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20240426 13:21:13.342448 30304 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.513294 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.514639 25101 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:13.513749 25098 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
W20240426 13:21:13.512850 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77661us
I20240426 13:21:13.516139 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.520730 25028 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:13.519342 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.519021 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.522305 25096 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.428751 30281 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.522599 25022 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:21:13.528338 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91749.1us
I20240426 13:21:13.401975 30187 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.530031 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.532104 25620 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:13.532539 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.517100 30201 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:13.503386 30193 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.543643 30313 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:13.547581 30011 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.522714 30255 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.549489 30179 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:13.556931 29956 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:13.559192 30188 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.561002 30255 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:13.562135 30164 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.500s	user 0.007s	sys 0.000s
I20240426 13:21:13.566359 30191 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.568451 30201 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:13.572201 30364 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20240426 13:21:13.574632 30178 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.546s	user 0.004s	sys 0.004s
I20240426 13:21:13.575487 25105 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20240426 13:21:13.575867 25105 pending_rounds.cc:76] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 10
I20240426 13:21:13.576117 25105 pending_rounds.cc:97] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20240426 13:21:13.576395 25105 raft_consensus.cc:2866] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:13.545612 30073 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:13.580570 30362 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.268s	user 0.000s	sys 0.000s
W20240426 13:21:13.582026 30365 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:21:13.584138 30313 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.584462 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2049) took 1159 ms. Trace:
I20240426 13:21:13.584348 30353 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.585639 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.586395 25097 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:13.587105 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.587695 25097 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:13.588102 30204 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.590052 30304 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:13.590410 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.591979 30184 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13.
I20240426 13:21:13.593458 25097 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:13.557021 30179 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:13.601773 24947 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:13.596335 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:13.535843 30253 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:13.602802 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:13.554391 30360 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20240426 13:21:13.604948 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:13.674835 24954 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:13.606026 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.598781 25619 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:13.617434 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.585108 25103 rpcz_store.cc:276] 0426 13:21:12.425567 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.505132 (+ 79565us) service_pool.cc:225] Handling call
0426 13:21:12.505420 (+   288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:12.712361 (+206941us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:12.712391 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:12.720208 (+  7817us) log.cc:842] Serialized 26 byte log entry
0426 13:21:13.136247 (+416039us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:13.136279 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:13.136414 (+   135us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:13.584186 (+447772us) raft_consensus.cc:1671] finished
0426 13:21:13.584215 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:13.584394 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51852,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":47,"thread_start_us":622310,"threads_started":2,"wal-append.queue_time_us":591774}
I20240426 13:21:13.618441 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:13.605988 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49598.2us
I20240426 13:21:13.676865 25615 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:13.677121 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.604290 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:13.678089 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 73913.9us
W20240426 13:21:13.678666 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61335.6us
I20240426 13:21:13.616205 30317 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:13.678679 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:13.679481 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.584563 30313 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:13.688688 30188 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.689118 30191 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:13.690634 29759 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:13.691282 29759 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:13.691787 29759 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:13.692817 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:13.692570 29759 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.693508 30199 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:13.695313 29956 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:13.617331 30253 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:13.696363 30253 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.697506 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.698561 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.679816 30351 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.700424 30351 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:13.681001 30187 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:13.702637 30148 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:13.703073 30148 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.554718 24850 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:13.705786 24850 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:13.689134 25615 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:13.681605 30193 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:13.749271 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72218.7us
I20240426 13:21:13.716718 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:13.751783 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.753418 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:13.757169 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:13.687474 30281 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:13.760839 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43619us
I20240426 13:21:13.763339 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:13.764858 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74438us
I20240426 13:21:13.767129 30281 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:13.763036 30304 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:13.761286 30154 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.752s	user 0.008s	sys 0.002s
I20240426 13:21:13.774881 25098 raft_consensus.cc:2354] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:13.775187 25589 leader_election.cc:398] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16.
I20240426 13:21:13.775523 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2417) took 1400 ms. Trace:
I20240426 13:21:13.776335 25096 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:13.775521 25589 leader_election.cc:401] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20240426 13:21:13.777361 25101 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:13.778584 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:13.779054 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62418.5us
I20240426 13:21:13.778577 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:13.780731 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.781503 25025 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.775911 25099 rpcz_store.cc:276] 0426 13:21:12.374876 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.491857 (+116981us) service_pool.cc:225] Handling call
0426 13:21:12.492235 (+   378us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:12.492588 (+   353us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:12.492626 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:12.659376 (+166750us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b2c0012e098. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:12.659711 (+   335us) log.cc:842] Serialized 26 byte log entry
0426 13:21:13.024076 (+364365us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:13.039912 (+ 15836us) log.cc:842] Serialized 14 byte log entry
0426 13:21:13.081320 (+ 41408us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:13.081414 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:13.775269 (+693855us) raft_consensus.cc:1671] finished
0426 13:21:13.775355 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:13.775474 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56698,"prepare.queue_time_us":17505,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":36,"raft.queue_time_us":113091,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":25779,"spinlock_wait_cycles":37144320,"thread_start_us":363962,"threads_started":1,"wal-append.queue_time_us":569962}
I20240426 13:21:13.782066 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.785188 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.788334 25101 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
W20240426 13:21:13.788681 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 102102us
I20240426 13:21:13.789414 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:13.796191 25096 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:13.803354 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.011s	sys 0.062s
W20240426 13:21:13.805002 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.011s	sys 0.062s
I20240426 13:21:13.806630 25025 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:13.813551 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:13.676508 30178 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:13.822664 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.858826 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:13.869493 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56037.7us
I20240426 13:21:13.870467 25097 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:13.707424 24850 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 9 to 10, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 10 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:13.761467 30294 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:13.877893 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.884575 25029 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:13.876127 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.888502 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:13.889868 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2047) took 1513 ms. Trace:
I20240426 13:21:13.890292 25105 rpcz_store.cc:276] 0426 13:21:12.376268 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.504637 (+128369us) service_pool.cc:225] Handling call
0426 13:21:12.504881 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:13.576832 (+1071951us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:13.889621 (+312789us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003b06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:13.889808 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2461197568}
I20240426 13:21:13.907424 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.812355 29956 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.796559 30301 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:13.796712 30304 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16.
I20240426 13:21:13.754976 30314 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:13.928951 25030 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.000s	sys 0.038s
I20240426 13:21:13.828284 30226 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:13.922340 30274 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:13.874018 30191 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:13.909816 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:13.946632 25030 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.000s	sys 0.038s
I20240426 13:21:13.944896 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.878561 30315 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:13.942500 25096 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:13.952574 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:13.788055 30253 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.909246 30334 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:13.995851 30193 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:13.996476 30193 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:13.997447 30193 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.000093 30148 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.001739 30253 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.002401 30315 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:14.003504 30315 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:13.943641 30372 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:13.909627 30188 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:14.009085 25104 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.017s	sys 0.011s
W20240426 13:21:14.009389 25104 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.017s	sys 0.011s
W20240426 13:21:13.990873 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46076.3us
I20240426 13:21:13.984160 25098 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:13.992982 25028 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:13.993165 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:14.013146 25096 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:14.013187 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:13.981029 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2076) took 1044 ms. Trace:
I20240426 13:21:13.886029 30178 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.014292 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:13.976922 30301 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:14.018188 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 919) took 1072 ms. Trace:
I20240426 13:21:14.016260 29429 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.022711 25028 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:14.015813 25029 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:14.030643 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:13.999347 25105 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:14.014681 25030 rpcz_store.cc:276] 0426 13:21:12.936327 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.936476 (+   149us) service_pool.cc:225] Handling call
0426 13:21:12.936845 (+   369us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:12.941516 (+  4671us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:12.941557 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:12.982361 (+ 40804us) log.cc:842] Serialized 52 byte log entry
0426 13:21:13.376160 (+393799us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:13.980440 (+604280us) log.cc:842] Serialized 14 byte log entry
0426 13:21:13.980698 (+   258us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:13.980774 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:13.980822 (+    48us) raft_consensus.cc:1671] finished
0426 13:21:13.980833 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:13.980976 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8507,"prepare.queue_time_us":624094,"raft.queue_time_us":8970,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":18408,"spinlock_wait_cycles":15127936,"thread_start_us":927891,"threads_started":2,"wal-append.queue_time_us":578459}
I20240426 13:21:14.029119 25620 rpcz_store.cc:276] 0426 13:21:12.945489 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.945606 (+   117us) service_pool.cc:225] Handling call
0426 13:21:14.018030 (+1072424us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:14.018139 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1257909504}
I20240426 13:21:13.994655 30334 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:13.946673 29956 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.045677 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.023203 30188 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.048348 25101 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:14.050956 25100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.029s	sys 0.021s
W20240426 13:21:14.053591 25100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.030s	sys 0.022s
I20240426 13:21:14.036651 30148 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:14.055145 25026 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.017s	sys 0.040s
I20240426 13:21:13.984737 30179 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:14.053893 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.024s	sys 0.028s
I20240426 13:21:13.984196 30274 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:13.992051 30191 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:14.057121 25026 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.017s	sys 0.040s
W20240426 13:21:14.062412 25616 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.010s	sys 0.001s
W20240426 13:21:14.062912 25616 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.011s	sys 0.001s
I20240426 13:21:14.063129 25098 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:21:14.059351 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.024s	sys 0.028s
I20240426 13:21:14.044137 29909 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:14.062288 25105 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:14.012423 30313 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:14.085413 30296 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s	user 0.021s	sys 0.016s
W20240426 13:21:14.063717 30340 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.011s	sys 0.002s
I20240426 13:21:14.040307 29429 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.064306 30204 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.119602 30204 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.121747 29956 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.122002 30204 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.123013 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.130759 25104 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20240426 13:21:14.132690 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 922) took 1158 ms (client timeout 1507 ms). Trace:
W20240426 13:21:14.133183 25619 rpcz_store.cc:269] 0426 13:21:12.974695 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.974817 (+   122us) service_pool.cc:225] Handling call
0426 13:21:14.132504 (+1157687us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:14.132625 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1000619392}
I20240426 13:21:14.135874 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.136237 30274 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.138334 30191 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.140985 30188 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.091228 25101 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
W20240426 13:21:14.093132 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 720375us
I20240426 13:21:14.059648 25096 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
W20240426 13:21:14.102926 30296 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s	user 0.021s	sys 0.016s
I20240426 13:21:14.022817 30372 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.115391 25097 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
W20240426 13:21:14.108579 30340 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.011s	sys 0.002s
I20240426 13:21:14.154002 24949 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:14.061512 30148 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:14.097499 30325 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.007s	sys 0.016s
I20240426 13:21:14.176733 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 703) took 1099 ms. Trace:
I20240426 13:21:14.179072 25027 rpcz_store.cc:276] 0426 13:21:13.077531 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.146087 (+ 68556us) service_pool.cc:225] Handling call
0426 13:21:13.186082 (+ 39995us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:13.186340 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:13.186375 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:13.237697 (+ 51322us) log.cc:842] Serialized 26 byte log entry
0426 13:21:13.807447 (+569750us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:13.807500 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:13.807585 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.176562 (+368977us) raft_consensus.cc:1671] finished
0426 13:21:14.176581 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.176681 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":517697,"threads_started":1,"wal-append.queue_time_us":807533}
I20240426 13:21:14.183008 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:14.183866 25623 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:14.142422 30314 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.108253 29909 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:14.076195 30334 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.199067 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:14.181385 30325 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.007s	sys 0.016s
I20240426 13:21:14.078593 30301 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:14.243157 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.015s	sys 0.058s
W20240426 13:21:14.243515 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.015s	sys 0.058s
I20240426 13:21:14.251340 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.203812 30178 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.235327 30334 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:14.286324 25623 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:14.287603 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:14.288340 25615 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:21:14.289459 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:14.290827 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:14.252792 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.292407 24949 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:14.263758 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.294873 24950 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:14.268826 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.295537 24947 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:14.269752 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.273964 30179 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.282373 29429 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.285122 30313 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.296582 24953 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:21:14.312394 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 441959us
I20240426 13:21:14.298708 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:14.313186 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:14.314477 24953 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20240426 13:21:14.321461 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.323040 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.323647 24947 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:14.348191 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.348848 24953 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:14.350072 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:14.361688 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:14.361533 30204 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.363572 30204 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.364965 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1027) took 1016 ms. Trace:
I20240426 13:21:14.365396 25616 rpcz_store.cc:276] 0426 13:21:13.348672 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.348845 (+   173us) service_pool.cc:225] Handling call
0426 13:21:13.360479 (+ 11634us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:13.360839 (+   360us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:13.360869 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:13.373171 (+ 12302us) log.cc:842] Serialized 26 byte log entry
0426 13:21:14.063425 (+690254us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:14.063531 (+   106us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.063618 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.364744 (+301126us) raft_consensus.cc:1671] finished
0426 13:21:14.364768 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.364899 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3113,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":12507,"thread_start_us":689935,"threads_started":1,"wal-append.queue_time_us":989857}
I20240426 13:21:14.367111 30253 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:14.367569 30253 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.367909 30253 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:14.371742 30188 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:14.371505 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:14.372828 30237 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.373521 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:14.374619 30148 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.375202 30148 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:14.375633 30148 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.375948 30148 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:14.374675 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.379966 30274 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:14.380708 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.385949 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2082) took 1141 ms. Trace:
I20240426 13:21:14.387310 30317 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.388190 29759 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.393158 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.373898 30193 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:14.320888 30325 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.394975 24947 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:14.412452 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.413137 25028 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:14.410123 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2062) took 1740 ms. Trace:
I20240426 13:21:14.396534 25026 rpcz_store.cc:276] 0426 13:21:13.244386 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.244869 (+   483us) service_pool.cc:225] Handling call
0426 13:21:13.245223 (+   354us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:21:13.253872 (+  8649us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:13.253911 (+    39us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:21:13.275163 (+ 21252us) log.cc:842] Serialized 78 byte log entry
0426 13:21:13.502995 (+227832us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:14.062402 (+559407us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.385324 (+322922us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b1800440da0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:21:14.385517 (+   193us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.385592 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.385638 (+    46us) raft_consensus.cc:1671] finished
0426 13:21:14.385730 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.385884 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46004,"prepare.queue_time_us":541309,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":332733,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":9121,"spinlock_wait_cycles":2631040,"thread_start_us":704566,"threads_started":2,"wal-append.queue_time_us":458356}
I20240426 13:21:14.400862 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.359537 30164 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:14.349730 29429 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.357259 30226 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:14.365527 30204 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.417023 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.439145 25025 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:21:14.439796 25027 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20240426 13:21:14.440181 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.374068 30237 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:14.425258 25030 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:14.275758 29909 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.428988 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:14.450481 25029 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 17
I20240426 13:21:14.409754 30182 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:14.431037 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:14.452785 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58672.1us
I20240426 13:21:14.453572 24996 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
W20240426 13:21:14.455032 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 311038us
I20240426 13:21:14.456881 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.437358 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.437812 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.455772 24996 leader_election.cc:398] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16.
I20240426 13:21:14.420576 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.459213 25026 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:14.442807 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:14.359109 30255 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20240426 13:21:14.410624 30188 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.462090 30188 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.450505 24949 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20240426 13:21:14.414152 25024 rpcz_store.cc:276] 0426 13:21:12.669890 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.752923 (+ 83033us) service_pool.cc:225] Handling call
0426 13:21:12.753291 (+   368us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:12.772191 (+ 18900us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:12.773452 (+  1261us) log.cc:842] Serialized 14 byte log entry
0426 13:21:13.789659 (+1016207us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:13.790041 (+   382us) log.cc:842] Serialized 26 byte log entry
0426 13:21:13.791311 (+  1270us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:14.193281 (+401970us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.409805 (+216524us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.409885 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.409932 (+    47us) raft_consensus.cc:1671] finished
0426 13:21:14.409947 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.410066 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":218974,"prepare.queue_time_us":285455,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":10958,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":17589,"thread_start_us":1417433,"threads_started":3,"wal-append.queue_time_us":745631}
I20240426 13:21:14.440279 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.428436 30193 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:14.479403 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:14.487980 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:14.444281 30164 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:14.405081 30317 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.447355 30021 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:14.490094 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.424921 30325 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.490520 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:14.447175 30165 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.446954 30294 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.498147 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2054) took 1152 ms. Trace:
I20240426 13:21:14.410339 29759 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.472796 30340 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.473261 30372 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.512977 24953 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:21:14.515858 25025 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 13:21:14.515926 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.517989 24947 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:14.498503 30380 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:14.439567 29429 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.512845 25100 rpcz_store.cc:276] 0426 13:21:13.345448 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.345587 (+   139us) service_pool.cc:225] Handling call
0426 13:21:13.345974 (+   387us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:13.368473 (+ 22499us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:13.368498 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:13.368970 (+   472us) log.cc:842] Serialized 26 byte log entry
0426 13:21:14.055575 (+686605us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:14.486109 (+430534us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.497766 (+ 11657us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.497860 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.497916 (+    56us) raft_consensus.cc:1671] finished
0426 13:21:14.497931 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.498069 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26083,"prepare.queue_time_us":333343,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":23509,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":9934,"thread_start_us":1031119,"threads_started":2,"wal-append.queue_time_us":859860}
I20240426 13:21:14.471999 30274 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20240426 13:21:14.515617 30165 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:14.532922 30391 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:21:14.513351 30021 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.534337 30165 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.535586 30021 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.536386 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.537060 24949 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:14.537946 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.538504 30241 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.538838 24955 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20240426 13:21:14.539036 30241 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.541452 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.542759 30241 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.540838 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.543387 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.543745 24952 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:14.544190 30359 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:14.544598 30359 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.544909 30359 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:14.545059 29759 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.548547 30193 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.532572 30379 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:14.553608 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.490155 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:14.555341 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65255.7us
I20240426 13:21:14.477684 30204 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20240426 13:21:14.564754 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.530251 24947 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:14.569051 30152 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:14.570307 30317 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.507050 30297 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20240426 13:21:14.571774 30188 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.573509 30148 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.575409 30148 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.523855 30178 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:14.576599 30373 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:14.577123 30373 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.578801 30373 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.563776 30379 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.581251 29429 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:14.582006 30379 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.583307 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2831) took 1056 ms. Trace:
I20240426 13:21:14.583667 25022 rpcz_store.cc:276] 0426 13:21:13.527098 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.527255 (+   157us) service_pool.cc:225] Handling call
0426 13:21:13.544120 (+ 16865us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:13.551171 (+  7051us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:13.551200 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:13.556166 (+  4966us) log.cc:842] Serialized 52 byte log entry
0426 13:21:14.085963 (+529797us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:14.099860 (+ 13897us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.409538 (+309678us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.409626 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:14.583115 (+173489us) raft_consensus.cc:1671] finished
0426 13:21:14.583132 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.583259 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":324572,"prepare.queue_time_us":13305,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":316529,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":19740,"thread_start_us":400982,"threads_started":1,"wal-append.queue_time_us":840485}
I20240426 13:21:14.557610 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:14.591054 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.592294 25102 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:14.594094 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2064) took 1021 ms. Trace:
I20240426 13:21:14.594534 25105 rpcz_store.cc:276] 0426 13:21:13.572817 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.041678 (+468861us) service_pool.cc:225] Handling call
0426 13:21:14.042154 (+   476us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:14.097228 (+ 55074us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.593847 (+496619us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b540009c4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:14.594031 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45533952}
I20240426 13:21:14.596065 25099 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:14.596287 30379 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:14.598158 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:14.587157 30154 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20240426 13:21:14.599815 30404 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.600342 30404 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.564095 30199 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:14.604025 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 214328us
I20240426 13:21:14.569008 30361 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:14.607761 25099 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20240426 13:21:14.608633 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.609182 25102 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:14.609877 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.610066 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.610483 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:14.610623 25097 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
W20240426 13:21:14.598286 30399 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20240426 13:21:14.623786 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 459785us
I20240426 13:21:14.573519 30164 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.601162 30152 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:14.597493 30336 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:14.523633 30340 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.654112 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.642522 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:14.516935 30294 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.660805 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.667402 25097 raft_consensus.cc:2354] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
W20240426 13:21:14.659307 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61197.8us
I20240426 13:21:14.661849 25105 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:14.662782 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:14.654875 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 303530us
I20240426 13:21:14.623834 30379 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.682881 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:14.681142 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.595767 30359 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.642109 30178 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:14.702661 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.710740 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:14.711462 25105 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:14.719143 25028 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:14.727761 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:14.668196 30325 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.673271 30165 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:14.671185 30154 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:14.650370 30317 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.070s
I20240426 13:21:14.701930 30315 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:14.744714 25100 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:14.652076 30297 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.071s
W20240426 13:21:14.748574 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46013.1us
I20240426 13:21:14.749162 25099 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:14.742053 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.752818 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.742852 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.779433 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.781312 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:14.777519 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:14.801267 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:14.778395 25102 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:14.716727 30336 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:14.650557 30380 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.100s
I20240426 13:21:14.706111 30406 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:14.789330 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.822026 25099 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:14.791180 25097 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:14.772480 30189 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:14.720147 30400 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.067s
I20240426 13:21:14.796988 30404 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.798169 30340 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.643888 30361 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.746572 29759 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:14.708564 30379 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.746189 30165 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.733379 30148 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
W20240426 13:21:14.800334 30388 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20240426 13:21:14.829532 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48331.5us
I20240426 13:21:14.829566 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 913) took 1016 ms. Trace:
I20240426 13:21:14.787804 30187 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:14.826136 30189 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:14.833740 30400 consensus_queue.cc:1225] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240426 13:21:14.839493 30340 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:14.841667 30361 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.842625 30359 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:14.843230 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2427) took 1242 ms. Trace:
I20240426 13:21:14.867892 25101 rpcz_store.cc:276] 0426 13:21:13.601282 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:13.812386 (+211104us) service_pool.cc:225] Handling call
0426 13:21:14.843059 (+1030673us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:14.843169 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366044416}
I20240426 13:21:14.835224 30178 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.843600 30334 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:14.821388 30315 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:14.845211 29759 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.830828 30406 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.800320 30073 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:14.782851 30382 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:14.856686 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:14.831283 30199 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:14.896941 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2426) took 1296 ms. Trace:
I20240426 13:21:14.865665 30294 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:14.897284 25098 rpcz_store.cc:276] 0426 13:21:13.601004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.064007 (+463003us) service_pool.cc:225] Handling call
0426 13:21:14.096935 (+ 32928us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:14.097122 (+   187us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:14.491228 (+394106us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.896544 (+405316us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:14.896603 (+    59us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:14.896640 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.896702 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.896893 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98,"prepare.queue_time_us":292024,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"thread_start_us":767736,"threads_started":2}
I20240426 13:21:14.899335 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.898820 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:14.866380 30340 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:14.855350 30165 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:14.835688 30021 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20240426 13:21:14.864166 30189 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:14.909802 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:14.910068 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:14.910406 24955 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:14.910779 25622 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 12
I20240426 13:21:14.876907 29759 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:14.898780 30274 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:14.897985 30359 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.901515 30148 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:14.870592 30334 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.871975 30187 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:14.903679 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:14.884953 30315 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.934473 30187 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.935070 30334 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:14.939323 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:14.946733 24947 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20240426 13:21:14.952735 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 269901us
I20240426 13:21:14.941175 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:14.955117 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.896262 30073 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.956218 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:14.957051 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:14.968096 25622 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:14.857684 25096 rpcz_store.cc:276] 0426 13:21:13.813642 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.163137 (+349495us) service_pool.cc:225] Handling call
0426 13:21:14.163442 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:14.163649 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:14.530826 (+367177us) log.cc:842] Serialized 14 byte log entry
0426 13:21:14.829259 (+298433us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:14.829307 (+    48us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:14.829367 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.829425 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.829518 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12265,"prepare.queue_time_us":225652,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":22,"raft.queue_time_us":7218,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":649137,"threads_started":2}
I20240426 13:21:14.884608 30336 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:14.930662 30148 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:14.994299 25101 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:15.005476 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:14.995085 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:14.996451 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:15.007705 25025 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:15.008222 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 259188us
I20240426 13:21:14.999326 25029 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:14.902767 30199 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:14.993536 25097 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:15.030771 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.032392 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:15.035662 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.011710 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:14.930907 30359 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.009176 30193 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:15.007717 30148 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:15.049232 25101 pending_rounds.cc:76] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 7
I20240426 13:21:15.050367 25101 pending_rounds.cc:97] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20240426 13:21:15.050684 25101 raft_consensus.cc:2866] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:15.006124 30431 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:15.060796 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.048558 25031 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:15.052625 25589 leader_election.cc:398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
I20240426 13:21:15.065618 25589 leader_election.cc:401] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:15.067169 30361 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
I20240426 13:21:15.067301 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:15.073879 30361 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:15.074337 30361 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:15.074779 30361 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:15.077630 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.079864 25025 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:15.080739 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:15.082449 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.084712 25031 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:15.054841 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:15.086391 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 428068us
I20240426 13:21:15.089841 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:15.090806 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:15.027395 30317 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:15.031265 30380 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.092406 25616 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:21:15.095109 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.095950 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:15.099052 25616 raft_consensus.cc:595] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:15.096181 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.043131 30154 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:15.108695 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.110064 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.112829 25097 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:15.113005 24995 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:15.113354 24995 leader_election.cc:401] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20240426 13:21:15.114104 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.114795 24947 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:14.945303 30340 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.121605 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.049748 30379 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.119046 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:15.133467 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:15.135563 24948 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:15.136212 24955 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:15.136484 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 177982us
I20240426 13:21:15.093180 30412 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:15.140652 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.087776 30424 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.155088 30193 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:15.129462 30237 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.677s	user 0.009s	sys 0.001s
W20240426 13:21:15.150619 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 188832us
I20240426 13:21:15.167719 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.044351 30432 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
I20240426 13:21:15.128424 30406 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.195792 30412 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:15.169098 24950 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:15.170395 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.206643 24948 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20240426 13:21:15.189358 30253 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.821s	user 0.005s	sys 0.005s
I20240426 13:21:15.140872 30011 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:15.188812 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.116168 30317 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.188102 30340 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:15.168457 30440 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:15.119931 30380 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.097506 30431 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.215062 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.235397 30317 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.221868 30253 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.242065 24955 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:15.141427 30441 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.096s
I20240426 13:21:15.249203 25097 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:21:15.162957 30193 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.191738 30125 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20240426 13:21:15.256723 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:15.205099 30410 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:15.249557 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 136688us
W20240426 13:21:15.251191 30414 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.294s	user 0.000s	sys 0.001s
I20240426 13:21:15.214224 30011 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:15.321805 30011 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:15.193857 30400 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20240426 13:21:15.232460 30304 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.326469 30304 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.157487 30424 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.328939 30304 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.310004 25025 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:15.329622 30424 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.302150 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:15.330413 30443 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:15.330857 30443 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:15.245548 30440 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:15.339068 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.314958 24951 raft_consensus.cc:2354] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
I20240426 13:21:15.302186 30315 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.343222 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:15.320861 30435 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:21:15.253453 30187 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.338761 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.352609 24954 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:15.339721 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:15.353137 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.354156 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:15.355293 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:15.350831 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94193.8us
I20240426 13:21:15.336022 30073 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.430625 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.344117 30440 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:15.310746 30361 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.421705 30359 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.331597 30443 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.427450 30410 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:15.427263 30372 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:15.451709 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2236) took 1299 ms. Trace:
I20240426 13:21:15.452123 24956 rpcz_store.cc:276] 0426 13:21:14.152670 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.152789 (+   119us) service_pool.cc:225] Handling call
0426 13:21:14.153092 (+   303us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:14.174446 (+ 21354us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:14.174477 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:14.176920 (+  2443us) log.cc:842] Serialized 26 byte log entry
0426 13:21:14.379801 (+202881us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b180029d7c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:14.680887 (+301086us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:14.680927 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.680990 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:15.443312 (+762322us) raft_consensus.cc:1671] finished
0426 13:21:15.443334 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:15.451644 (+  8310us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1845,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":13600,"spinlock_wait_cycles":5212928,"thread_start_us":300883,"threads_started":1,"wal-append.queue_time_us":725476}
I20240426 13:21:15.356235 30406 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.090s
I20240426 13:21:15.440448 30199 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.440162 30198 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.450376 30201 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:15.472020 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 476953us
I20240426 13:21:15.451236 30410 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:15.448868 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.452049 30372 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:15.480688 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:15.482141 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.486712 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.488289 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:15.487215 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.466174 30443 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:15.504925 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.508729 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2751) took 1078 ms. Trace:
I20240426 13:21:15.509156 24950 rpcz_store.cc:276] 0426 13:21:14.430545 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.430765 (+   220us) service_pool.cc:225] Handling call
0426 13:21:15.508539 (+1077774us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54000be854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:15.508656 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1646841600}
I20240426 13:21:15.464861 30456 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:15.483729 30410 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:15.520355 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2435) took 1825 ms. Trace:
I20240426 13:21:15.519348 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:21:15.501178 30439 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20240426 13:21:15.356478 30412 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.524607 25101 raft_consensus.cc:595] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.495359 30372 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:15.524312 25104 rpcz_store.cc:276] 0426 13:21:13.694864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.131668 (+436804us) service_pool.cc:225] Handling call
0426 13:21:14.157919 (+ 26251us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:14.180852 (+ 22933us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:14.180883 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:14.180932 (+    49us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:14.180970 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:14.181031 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:14.885540 (+704509us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18003cb320. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654
0426 13:21:15.520206 (+634666us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b54003e12d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:15.520307 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38090,"raft.queue_time_us":22992,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":63097984}
I20240426 13:21:15.494962 30436 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:15.526715 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 169174us
I20240426 13:21:15.530366 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.545814 30466 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240426 13:21:15.551872 30446 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
W20240426 13:21:15.552878 30437 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:21:15.554942 30334 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.000s
W20240426 13:21:15.557440 30444 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.223s	user 0.001s	sys 0.001s
W20240426 13:21:15.557571 30453 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240426 13:21:15.518975 30431 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.541479 30436 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:15.564615 30454 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20240426 13:21:15.567905 30380 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.569046 30361 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.571327 30372 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:15.575647 30165 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.662s	user 0.008s	sys 0.000s
I20240426 13:21:15.577112 29688 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:15.518544 30458 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:15.520606 30456 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:15.471168 30198 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.485059 30201 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.591449 30199 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.598402 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.599520 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.682528 25100 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:15.596024 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:15.589406 30189 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.673s	user 0.007s	sys 0.003s
W20240426 13:21:15.558156 30442 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.206s	user 0.000s	sys 0.000s
I20240426 13:21:15.663012 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 952) took 1206 ms. Trace:
I20240426 13:21:15.543666 30253 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.664243 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.664664 30433 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:15.590260 30179 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:15.666038 29759 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.741s	user 0.010s	sys 0.000s
W20240426 13:21:15.663879 30148 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.000s
I20240426 13:21:15.684612 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:15.682353 25105 raft_consensus.cc:2362] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13. Re-sending same reply.
W20240426 13:21:15.709951 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114018us
I20240426 13:21:15.670835 30441 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:15.710908 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:15.677722 30165 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.709566 24951 rpcz_store.cc:276] 0426 13:21:14.456340 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.456584 (+   244us) service_pool.cc:225] Handling call
0426 13:21:15.662767 (+1206183us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:15.662933 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1971152768}
I20240426 13:21:15.739528 25616 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:15.724196 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:15.741560 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:15.743232 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:15.716641 25023 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:15.679941 30456 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:15.670120 30469 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.068s
I20240426 13:21:15.727344 30198 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.727999 30467 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:15.728456 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43948us
I20240426 13:21:15.729241 30182 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:15.751953 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:15.729378 30201 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.712433 30464 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:15.714461 30380 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:15.731858 30011 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.711828 30189 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.732848 30148 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.733386 29759 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.663785 30334 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.741434 25588 leader_election.cc:398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12.
I20240426 13:21:15.752127 29446 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:15.770638 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 259823us
I20240426 13:21:15.710398 30199 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.771147 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.772409 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:15.726100 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 223027us
I20240426 13:21:15.759370 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.761170 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.764405 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.775053 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.755280 30431 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:15.753276 30182 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:15.773384 25100 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:15.754142 30475 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:15.791024 25029 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:15.739225 30441 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.783540 25027 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:15.764743 30469 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:15.759085 30467 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.793977 30334 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.794061 30148 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.794512 30482 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:15.794678 30201 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:15.812273 30201 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.796514 30152 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:15.813359 30152 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.814188 30201 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:15.814579 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.021s	sys 0.012s
W20240426 13:21:15.814872 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.021s	sys 0.012s
W20240426 13:21:15.815099 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 723395us
I20240426 13:21:15.815142 30152 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:15.797762 30165 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.779815 30182 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:15.818523 30410 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.819643 25622 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:15.800395 30297 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:15.820472 30297 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:15.800420 30372 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.821393 30297 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.761526 30464 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.822355 30481 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:15.811628 30482 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.822649 30297 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:15.823040 30481 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:15.824638 30340 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.000s
I20240426 13:21:15.770433 30380 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:15.800566 29759 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.786062 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:15.776626 30458 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:15.837227 25616 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:15.839623 30198 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:15.840070 30380 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:15.840747 30011 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.841041 30380 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.822255 30271 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:15.842160 30410 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:15.842432 30271 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.842891 30011 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:15.843367 30011 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.780174 30431 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:15.846230 30271 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:15.846177 30458 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.850554 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.842346 30199 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:15.852528 30474 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:21:15.853812 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.801545 30440 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:15.853827 30189 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:15.861790 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109923us
I20240426 13:21:15.862071 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.863701 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.864715 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.869658 25097 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:15.870481 25102 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:15.862813 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.871649 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.858882 30431 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:15.874720 30182 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:15.809039 30469 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:15.884136 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:15.884936 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.892489 25097 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:15.856681 30148 raft_consensus.cc:2741] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20240426 13:21:15.886183 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:15.888597 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:15.890317 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 136323us
I20240426 13:21:15.899902 25098 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:15.900235 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:15.891009 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 112447us
I20240426 13:21:15.897496 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:15.897730 24955 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:15.901304 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:15.907629 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:15.908212 24951 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:15.908943 25100 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:15.911811 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.912552 25102 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:15.956152 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:15.779959 30477 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.118s
I20240426 13:21:15.957646 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:15.958595 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61351.7us
I20240426 13:21:15.882151 30073 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:16.026346 25102 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:16.031821 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2847) took 1619 ms. Trace:
I20240426 13:21:16.032287 25028 rpcz_store.cc:276] 0426 13:21:14.415387 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.419952 (+  4565us) service_pool.cc:225] Handling call
0426 13:21:16.031662 (+1611710us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54000569d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:16.031768 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":594419584}
I20240426 13:21:16.032742 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.033046 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.033367 24952 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:16.033425 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:16.034289 25028 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:16.035055 25026 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:16.031837 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:16.029017 30164 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:16.036541 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:16.037919 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.038661 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:16.039121 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.039922 25619 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:16.039985 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.045295 24954 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:16.036654 30164 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:16.058036 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 285738us
I20240426 13:21:16.066689 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.060420 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:16.060873 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 100853us
I20240426 13:21:16.067462 25029 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:16.024909 24951 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:16.068584 24996 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:16.069059 24996 leader_election.cc:401] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:16.073076 25097 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
W20240426 13:21:16.027571 30474 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20240426 13:21:16.075501 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:15.957799 30359 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.076104 25028 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:16.076253 25102 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:16.094647 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:16.099185 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.102855 25024 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20240426 13:21:16.073279 30447 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20240426 13:21:16.107767 25029 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:16.111589 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:16.112512 25097 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:16.110186 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:16.066514 30165 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.070343 30410 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:16.136415 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.074293 30164 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:16.124858 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.089948 30379 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:21:16.149096 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 247839us
I20240426 13:21:16.103498 30294 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.152477 25028 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:16.149211 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:16.155731 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:16.158090 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 383083us
I20240426 13:21:16.159485 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:16.161454 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.162248 24950 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:16.165120 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:16.166831 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:16.157927 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:16.168145 25097 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:16.165864 24952 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:16.174031 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:16.089890 30359 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.148643 29688 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:16.179499 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.159155 30379 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:16.184448 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.158668 30467 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.166469 30436 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.211732 25098 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240426 13:21:16.211634 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.212733 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.142401 30482 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.171538 30201 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:16.217988 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.219049 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:16.220008 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:16.252560 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 153482us
I20240426 13:21:16.253340 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:16.142923 30165 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.181180 30204 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:16.172995 30464 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.181572 30470 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:16.162621 30294 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.185045 30410 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:16.222474 30217 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.214219 30334 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:16.279006 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 710) took 1138 ms. Trace:
I20240426 13:21:16.214161 30379 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:16.280016 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 708) took 1142 ms. Trace:
W20240426 13:21:16.263027 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49496.9us
I20240426 13:21:16.272260 30182 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.214708 30189 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.219600 30404 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
W20240426 13:21:16.361943 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77719.4us
I20240426 13:21:16.213115 30359 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:16.273656 30488 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.366s	user 0.001s	sys 0.000s
I20240426 13:21:16.264004 30463 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.403467 25025 rpcz_store.cc:276] 0426 13:21:15.140308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.140423 (+   115us) service_pool.cc:225] Handling call
0426 13:21:16.278807 (+1138384us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54002ae254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:16.278950 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":374882176}
W20240426 13:21:16.398156 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 180422us
I20240426 13:21:16.357926 30436 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:16.419981 25028 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:16.263381 30458 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.272449 30271 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:16.422449 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.268164 30493 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:16.263569 30507 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:16.263911 30441 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.278570 30204 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:16.422358 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:16.272430 30451 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240426 13:21:16.271549 30502 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
W20240426 13:21:16.271062 30500 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20240426 13:21:16.272565 30462 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:16.272063 30492 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20240426 13:21:16.278373 30165 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.273102 30011 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.403126 25031 rpcz_store.cc:276] 0426 13:21:15.138101 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.138254 (+   153us) service_pool.cc:225] Handling call
0426 13:21:15.164427 (+ 26173us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:15.235839 (+ 71412us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:15.235894 (+    55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:15.235934 (+    40us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:15.235985 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.279796 (+1043811us) spinlock_profiling.cc:243] Waited 71.2 ms on lock 0x7b5400415ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.279855 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:16.279966 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163801472}
W20240426 13:21:16.488060 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 65500.8us
I20240426 13:21:16.272861 30430 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.430866 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:16.417851 30334 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.369475 30410 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:16.434206 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 264620us
I20240426 13:21:16.441668 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.404906 30440 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.424587 30470 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.423564 30201 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:16.427069 30313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.000s	sys 0.013s
I20240426 13:21:16.427973 30382 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:16.428645 30404 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:16.483376 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 414954us
W20240426 13:21:16.585140 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 368376us
W20240426 13:21:16.439975 30380 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.008s	sys 0.016s
I20240426 13:21:16.434327 30507 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:16.272401 30480 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.338s
I20240426 13:21:16.413929 30217 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:16.417320 30189 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.484633 30436 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:16.490134 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.444969 30204 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:16.491374 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:16.272094 30459 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:16.498739 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 81108.2us
I20240426 13:21:16.508714 25097 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20240426 13:21:16.515633 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74038us
I20240426 13:21:16.521077 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:16.520464 25031 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20240426 13:21:16.451306 30359 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.572824 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:16.497634 30475 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.427723 30463 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.513722 30410 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:16.586972 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.585544 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.503836 30430 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.712488 25031 pending_rounds.cc:76] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 9
I20240426 13:21:16.657074 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:16.582286 30313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.000s	sys 0.013s
I20240426 13:21:16.572986 30462 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.545423 29688 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.574903 30477 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.546284 30441 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:16.637099 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2098) took 1597 ms. Trace:
I20240426 13:21:16.713605 25031 pending_rounds.cc:97] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20240426 13:21:16.643448 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.659009 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.772820 25031 raft_consensus.cc:2866] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:16.712815 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 222807us
I20240426 13:21:16.632324 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:16.593451 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:16.721542 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:16.651525 30443 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.132s	user 0.007s	sys 0.003s
W20240426 13:21:16.636191 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.012s	sys 0.022s
W20240426 13:21:16.842291 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.931s	user 0.012s	sys 0.032s
W20240426 13:21:16.728225 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56778.5us
W20240426 13:21:16.597182 30380 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.009s	sys 0.016s
W20240426 13:21:16.598934 30456 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.844s	user 0.009s	sys 0.000s
I20240426 13:21:16.584893 30195 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.729202 30179 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:16.589442 30271 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:16.635771 30517 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20240426 13:21:16.846402 30154 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.690s	user 0.011s	sys 0.014s
I20240426 13:21:16.741772 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:16.599486 30501 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:16.904567 30501 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:16.905627 30501 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:16.641275 30506 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.472s	user 0.002s	sys 0.000s
W20240426 13:21:16.636946 30524 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:21:16.578887 30165 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.908402 30165 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.909086 30271 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.910187 30165 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.774767 25025 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:16.776005 24952 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20240426 13:21:16.911291 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2460) took 1019 ms. Trace:
I20240426 13:21:16.772720 25024 rpcz_store.cc:276] 0426 13:21:15.039579 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.040142 (+   563us) service_pool.cc:225] Handling call
0426 13:21:15.040500 (+   358us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.103331 (+1062831us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.636825 (+533494us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54004164d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.637038 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2442173184}
I20240426 13:21:16.850183 30336 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.776116 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:16.922282 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2451) took 1382 ms. Trace:
I20240426 13:21:16.922782 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:16.588363 30470 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.621600 30497 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:16.801515 30294 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.777036 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:16.917232 30154 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.738s	user 0.017s	sys 0.015s
I20240426 13:21:16.641108 30217 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:16.625816 30182 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:16.805035 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 161676us
I20240426 13:21:16.800045 30334 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:16.796898 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 933) took 1315 ms. Trace:
I20240426 13:21:16.939427 25622 rpcz_store.cc:276] 0426 13:21:15.481821 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.482004 (+   183us) service_pool.cc:225] Handling call
0426 13:21:15.482360 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:15.820011 (+337651us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.796699 (+976688us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.796844 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":775161600}
I20240426 13:21:16.784623 30382 raft_consensus.cc:979] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:21:16.780131 30441 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.944772 30315 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:16.945430 30315 consensus_queue.cc:1225] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:21:16.945530 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2102) took 1114 ms. Trace:
I20240426 13:21:16.946316 30441 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.815129 30189 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.947320 30470 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.946044 25029 rpcz_store.cc:276] 0426 13:21:15.831118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.168248 (+337130us) service_pool.cc:225] Handling call
0426 13:21:16.183218 (+ 14970us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.279867 (+ 96649us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:16.280135 (+   268us) log.cc:842] Serialized 14 byte log entry
0426 13:21:16.486827 (+206692us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:16.486892 (+    65us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:16.486935 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.945312 (+458377us) spinlock_profiling.cc:243] Waited 96.4 ms on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.945362 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:16.945478 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11541,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":1464,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":11181,"spinlock_wait_cycles":251608320,"thread_start_us":206339,"threads_started":1,"wal-append.queue_time_us":302380}
W20240426 13:21:16.802795 30431 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.896s	user 0.009s	sys 0.000s
I20240426 13:21:16.828935 30430 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.600217 30507 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:16.653556 30164 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.829941 30477 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.732509 30518 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.832541 30304 raft_consensus.cc:979] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:16.827608 29688 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.728281 30475 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.731990 30201 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.937209 30154 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:16.769292 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 183830us
I20240426 13:21:16.963222 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.813792 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2443) took 2205 ms. Trace:
I20240426 13:21:16.963837 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2253) took 1590 ms. Trace:
W20240426 13:21:16.839972 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100641us
I20240426 13:21:16.818140 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:16.965236 24955 rpcz_store.cc:276] 0426 13:21:15.373212 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.373374 (+   162us) service_pool.cc:225] Handling call
0426 13:21:15.373788 (+   414us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:15.909257 (+535469us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.963582 (+1054325us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540038f4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.963768 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1204414848}
I20240426 13:21:16.599895 30508 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:16.851006 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:16.966622 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.970239 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:16.970647 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119752us
I20240426 13:21:16.592296 30193 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.873745 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.873965 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.729041 30352 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:16.635160 30510 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20240426 13:21:16.884577 24956 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:16.884243 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:16.884456 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:16.872305 29429 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:16.876878 30195 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.795419 30514 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:16.890843 30398 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:16.778872 30463 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.875342 30456 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.774052 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.911676 25096 rpcz_store.cc:276] 0426 13:21:15.891753 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.891921 (+   168us) service_pool.cc:225] Handling call
0426 13:21:15.892280 (+   359us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:15.892483 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:15.892508 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:15.892870 (+   362us) log.cc:842] Serialized 26 byte log entry
0426 13:21:16.037958 (+145088us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:16.038013 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.038120 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:16.640532 (+602412us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:16.911017 (+270485us) raft_consensus.cc:1671] finished
0426 13:21:16.911040 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:16.911237 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7702272,"thread_start_us":144801,"threads_started":1,"wal-append.queue_time_us":221437}
I20240426 13:21:16.803187 30204 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.911690 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:16.922808 25104 rpcz_store.cc:276] 0426 13:21:15.539875 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.540075 (+   200us) service_pool.cc:225] Handling call
0426 13:21:15.616008 (+ 75933us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:15.716490 (+100482us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:15.716518 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:15.716562 (+    44us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:15.716593 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.922024 (+1205431us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.922083 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:16.922220 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230607744}
I20240426 13:21:16.871692 30281 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.802064 30440 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:16.884125 30073 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.057842 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:16.916215 30191 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.058259 30073 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.058730 30191 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.059633 30281 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.060200 30073 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.061744 25030 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:17.061676 30191 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:16.943416 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 2.155s (SENT) [suppressed 1 similar messages]
W20240426 13:21:16.793155 30490 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.217s	user 0.000s	sys 0.000s
W20240426 13:21:16.949746 30413 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.112s	user 0.008s	sys 0.012s
I20240426 13:21:17.034971 29429 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.935818 30497 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.065225 29429 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.065585 30497 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:16.953662 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:17.066509 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 112932us
I20240426 13:21:17.067878 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:16.727927 30459 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:17.070570 30459 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:16.730129 30359 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.638051 30404 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.072310 30521 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:17.072829 30521 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.955017 30334 raft_consensus.cc:1067] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:21:16.945950 30315 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:17.074744 30521 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.075058 30315 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:17.075673 30512 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.584s	user 0.002s	sys 0.000s
I20240426 13:21:17.075861 30315 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.077069 30315 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:16.969579 24951 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:16.964076 25103 rpcz_store.cc:276] 0426 13:21:14.607960 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:14.608468 (+   508us) service_pool.cc:225] Handling call
0426 13:21:14.608726 (+   258us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:14.952401 (+343675us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:14.952444 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:16.029809 (+1077365us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b2c0012e098. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:16.030515 (+   706us) log.cc:842] Serialized 52 byte log entry
0426 13:21:16.090264 (+ 59749us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:16.090338 (+    74us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.813348 (+723010us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:16.813430 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:16.813496 (+    66us) raft_consensus.cc:1671] finished
0426 13:21:16.813515 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:16.813707 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61139,"raft.queue_time_us":55649,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":22975,"spinlock_wait_cycles":156858112,"thread_start_us":294589,"threads_started":2,"wal-append.queue_time_us":229440}
I20240426 13:21:16.931849 30336 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:16.756304 30462 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.080209 30336 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:17.019685 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 256437us
I20240426 13:21:17.081468 30462 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.078382 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.084378 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.085124 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.026723 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:17.037031 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:17.022429 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:17.086468 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:16.965879 30409 raft_consensus.cc:2741] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
I20240426 13:21:16.968673 30449 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:17.091423 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:17.050958 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:17.000993 30432 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:16.952486 30182 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:16.978302 30430 raft_consensus.cc:1067] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:16.957572 30431 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.013216 30380 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:16.999136 30508 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.063200 30195 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.068652 30440 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:17.071781 24954 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:17.076946 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 707) took 1173 ms. Trace:
I20240426 13:21:17.025658 30493 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:17.041833 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.090545 25101 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:17.092813 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:16.976874 30317 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:17.094251 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9911e0422347463c8781b2574c023744"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:17.057798 30484 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:17.104364 30379 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.677s	user 0.010s	sys 0.000s
W20240426 13:21:17.096737 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 322724us
I20240426 13:21:17.117022 30521 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:17.117883 30154 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.025463 30340 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:17.053346 30398 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.076859 30301 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:17.101667 25099 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.031s	sys 0.098s
I20240426 13:21:17.102145 24948 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:17.019774 30165 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20240426 13:21:17.078366 30204 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.117626 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:17.202432 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84875.8us
I20240426 13:21:17.203469 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:17.059255 30456 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:17.135293 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45302.5us
I20240426 13:21:17.010951 30193 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.216552 30193 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:17.147648 25103 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:17.217185 30477 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.217865 30164 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.195840 30493 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:17.219564 30493 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.222955 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.137295 25620 rpcz_store.cc:276] 0426 13:21:15.903708 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.903911 (+   203us) service_pool.cc:225] Handling call
0426 13:21:15.904229 (+   318us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:15.904428 (+   199us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:15.904452 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:15.931368 (+ 26916us) log.cc:842] Serialized 26 byte log entry
0426 13:21:16.172620 (+241252us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:16.172654 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.172715 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:16.954922 (+782207us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:17.076579 (+121657us) raft_consensus.cc:1671] finished
0426 13:21:17.076613 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.076882 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68534656,"thread_start_us":133325,"threads_started":1,"wal-append.queue_time_us":476025}
I20240426 13:21:17.148548 25097 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20240426 13:21:17.149653 25096 raft_consensus.cc:595] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:17.226435 25096 raft_consensus.cc:603] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:17.151417 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:17.098630 30432 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:17.150867 25027 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
W20240426 13:21:17.100499 30510 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:21:17.247802 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:17.095909 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58971.3us
I20240426 13:21:17.271983 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:16.958761 30518 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.100144 30514 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:16.999768 30188 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20240426 13:21:17.189433 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.190182 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.115595 30530 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:17.019678 30352 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.113288 30191 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:17.199177 25099 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.032s	sys 0.098s
I20240426 13:21:17.213629 30456 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:17.214452 30515 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:17.214846 30470 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:17.198796 30521 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:17.223098 30400 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:17.247766 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.249118 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:17.250504 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:17.216502 30475 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.157001 30359 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.272032 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:17.184288 30340 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.320329 24956 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20240426 13:21:17.139330 30412 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.085s
I20240426 13:21:17.355916 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:17.451546 25096 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:17.181418 30379 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.101074 30449 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.205380 30281 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:17.370594 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54976.3us
I20240426 13:21:17.374421 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.273783 30432 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:17.383446 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.385058 24952 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:17.389165 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:17.315266 30522 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.137284 30380 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.402467 25097 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:17.350116 30336 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:17.405589 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:17.359679 30448 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.003s	sys 0.015s
W20240426 13:21:17.449476 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 199082us
I20240426 13:21:17.449833 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:17.520826 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:17.199364 30484 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:17.486007 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 542587us
I20240426 13:21:17.456094 25103 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:17.531803 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.359436 30317 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:17.467926 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78819us
I20240426 13:21:17.370353 30426 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:17.370015 30456 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:17.448045 30271 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.005s	sys 0.014s
I20240426 13:21:17.367424 30462 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:17.385515 30431 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.396315 30073 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:17.392179 29688 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.487643 30477 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.523384 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:17.524963 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.604413 25097 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:17.464658 30486 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.004s	sys 0.016s
I20240426 13:21:17.454362 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1052) took 1608 ms. Trace:
I20240426 13:21:17.521440 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:17.539623 25023 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:17.539986 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:17.545786 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.611634 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.609256 25616 rpcz_store.cc:276] 0426 13:21:15.845744 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.845856 (+   112us) service_pool.cc:225] Handling call
0426 13:21:15.846138 (+   282us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:15.855267 (+  9129us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:16.036065 (+180798us) log.cc:842] Serialized 14 byte log entry
0426 13:21:16.179926 (+143861us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:16.211018 (+ 31092us) log.cc:842] Serialized 26 byte log entry
0426 13:21:16.211148 (+   130us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:16.211180 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.211245 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:17.454166 (+1242921us) raft_consensus.cc:1671] finished
0426 13:21:17.454187 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.454312 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15142,"prepare.queue_time_us":408067,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":35402,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":32205,"thread_start_us":317334,"threads_started":2,"wal-append.queue_time_us":380851}
W20240426 13:21:17.615121 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91825us
I20240426 13:21:17.467335 30530 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:17.386108 30191 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.500770 30475 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:21:17.522027 30448 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.004s	sys 0.015s
I20240426 13:21:17.550498 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:17.511724 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 377728us
I20240426 13:21:17.566783 30469 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.568199 30521 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.570248 30352 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:17.572652 30449 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:17.583015 30410 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.851s	user 0.008s	sys 0.000s
I20240426 13:21:17.585013 30379 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.588277 30380 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.588618 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:17.675447 25623 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:17.593463 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.594388 30182 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.509032 25096 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:17.683893 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.607957 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.571769 30484 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:17.613554 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.517908 30336 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.541922 30281 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
I20240426 13:21:17.612780 30511 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.606246 30406 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.651090 30475 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:17.648566 30201 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:17.557477 30317 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.500658 30522 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.563158 30514 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.682178 30469 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:17.612650 30477 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.700353 30372 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:17.555298 30426 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.600630 30462 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.690795 30410 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.660830 30432 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:17.666334 30456 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:17.606266 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.000s	sys 0.032s
I20240426 13:21:17.537084 24952 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:17.561094 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:17.546682 25022 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.003s	sys 0.016s
I20240426 13:21:17.579097 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2093) took 2580 ms. Trace:
I20240426 13:21:17.657145 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.084390 30314 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:17.944248 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2101) took 2113 ms (client timeout 1931 ms). Trace:
I20240426 13:21:18.084455 30398 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:17.714601 30475 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:17.979049 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 934) took 1049 ms. Trace:
I20240426 13:21:17.927145 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2774) took 1038 ms. Trace:
I20240426 13:21:17.927534 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2777) took 1010 ms. Trace:
I20240426 13:21:17.948797 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2879) took 1808 ms. Trace:
I20240426 13:21:17.976912 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 929) took 1104 ms. Trace:
I20240426 13:21:17.919530 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2258) took 1795 ms. Trace:
I20240426 13:21:17.943672 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2113) took 1435 ms. Trace:
I20240426 13:21:17.980219 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2088) took 2020 ms. Trace:
I20240426 13:21:17.947225 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2876) took 1880 ms. Trace:
I20240426 13:21:17.950510 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2107) took 2048 ms. Trace:
W20240426 13:21:17.945864 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2865) took 2423 ms (client timeout 1618 ms). Trace:
W20240426 13:21:17.973378 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2083) took 2113 ms (client timeout 2734 ms). Trace:
I20240426 13:21:17.948693 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2106) took 2107 ms. Trace:
I20240426 13:21:17.999595 30073 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:18.004240 30317 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:17.791404 30513 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.303s	user 0.002s	sys 0.000s
I20240426 13:21:17.935726 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.022365 30416 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:17.977773 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:17.947382 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:17.926213 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:18.203008 25027 rpcz_store.cc:269] 0426 13:21:15.830234 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.866665 (+ 36431us) service_pool.cc:225] Handling call
0426 13:21:17.943698 (+2077033us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.943918 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2931482880}
W20240426 13:21:17.798825 30507 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.835s	user 0.007s	sys 0.000s
W20240426 13:21:17.784317 30531 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
W20240426 13:21:17.790158 30540 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20240426 13:21:18.216466 25028 rpcz_store.cc:276] 0426 13:21:16.066959 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.927295 (+860336us) service_pool.cc:225] Handling call
0426 13:21:16.936761 (+  9466us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.086204 (+149443us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:17.086234 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:17.086274 (+    40us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:17.086302 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.946651 (+860349us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400241154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.946723 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.946982 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":343246080}
W20240426 13:21:17.966799 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 500653us
W20240426 13:21:17.773409 30539 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20240426 13:21:17.812177 30525 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.169s	user 0.004s	sys 0.000s
I20240426 13:21:18.000336 30182 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:17.916273 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.003522 30336 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.080672 30433 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:17.981200 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:18.026017 30281 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:18.217514 25105 rpcz_store.cc:269] 0426 13:21:15.859716 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.861817 (+  2101us) service_pool.cc:225] Handling call
0426 13:21:17.972814 (+2110997us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.973120 (+   306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4093079424}
I20240426 13:21:18.216136 25102 rpcz_store.cc:276] 0426 13:21:15.962508 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.027703 (+ 65195us) service_pool.cc:225] Handling call
0426 13:21:16.076082 (+ 48379us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.083740 (+  7658us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.979717 (+1895977us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b5400169c50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.980005 (+   288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110403456}
I20240426 13:21:18.025456 30193 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.220227 25026 rpcz_store.cc:276] 0426 13:21:14.998327 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.004846 (+  6519us) service_pool.cc:225] Handling call
0426 13:21:16.034865 (+1030019us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.035453 (+   588us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.578897 (+1543444us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000569d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.579037 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2368136576}
I20240426 13:21:18.213243 24954 rpcz_store.cc:276] 0426 13:21:16.888204 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.028788 (+140584us) service_pool.cc:225] Handling call
0426 13:21:17.029120 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.136695 (+107575us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.926534 (+789839us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.926962 (+   428us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97375488}
I20240426 13:21:17.970599 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.217419 25023 rpcz_store.cc:276] 0426 13:21:15.841389 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.107893 (+266504us) service_pool.cc:225] Handling call
0426 13:21:17.948190 (+1840297us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.948490 (+   300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134203264}
I20240426 13:21:18.217150 25025 rpcz_store.cc:276] 0426 13:21:15.902203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.427321 (+525118us) service_pool.cc:225] Handling call
0426 13:21:16.427564 (+   243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.910781 (+483217us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.950055 (+1039274us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54000b0c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.950355 (+   300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797806336}
I20240426 13:21:18.215320 25024 rpcz_store.cc:276] 0426 13:21:16.140636 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.912107 (+771471us) service_pool.cc:225] Handling call
0426 13:21:16.912389 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.150271 (+237882us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:17.150315 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:17.150368 (+    53us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:17.150420 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.948343 (+797923us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.948419 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.948645 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":546505088}
I20240426 13:21:18.215385 25098 rpcz_store.cc:276] 0426 13:21:16.872632 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.872840 (+   208us) service_pool.cc:225] Handling call
0426 13:21:16.873149 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.065820 (+192671us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:17.065854 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:17.065903 (+    49us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:17.065940 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.976327 (+910387us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540009ced4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.976405 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.976668 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442679680}
W20240426 13:21:18.217190 25030 rpcz_store.cc:269] 0426 13:21:15.522240 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.530132 (+  7892us) service_pool.cc:225] Handling call
0426 13:21:17.945277 (+2415145us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.945605 (+   328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3367327104}
I20240426 13:21:18.212889 25101 rpcz_store.cc:276] 0426 13:21:16.929699 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.935256 (+  5557us) service_pool.cc:225] Handling call
0426 13:21:16.935545 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.147327 (+211782us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.978449 (+831122us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.978823 (+   374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346873088}
I20240426 13:21:17.620456 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.215703 24951 rpcz_store.cc:276] 0426 13:21:16.124372 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.124627 (+   255us) service_pool.cc:225] Handling call
0426 13:21:17.918891 (+1794264us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b540030c5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:17.919256 (+   365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1888457088}
I20240426 13:21:17.751547 30518 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:18.282577 30404 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.023s	sys 0.035s
I20240426 13:21:18.214622 24947 rpcz_store.cc:276] 0426 13:21:16.917283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.041896 (+124613us) service_pool.cc:225] Handling call
0426 13:21:17.048657 (+  6761us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:17.054071 (+  5414us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:17.054110 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:17.054492 (+   382us) log.cc:842] Serialized 26 byte log entry
0426 13:21:17.411265 (+356773us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:17.411295 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.923891 (+512596us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b5400078ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:17.923985 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:17.926787 (+  2802us) raft_consensus.cc:1671] finished
0426 13:21:17.926815 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.927352 (+   537us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11929856,"thread_start_us":356529,"threads_started":1,"wal-append.queue_time_us":689053}
I20240426 13:21:17.719794 30406 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:17.798640 30508 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:18.286175 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.018s	sys 0.047s
I20240426 13:21:17.743662 30201 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:18.284806 25029 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:17.795186 30522 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.215852 25031 rpcz_store.cc:276] 0426 13:21:16.507975 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.923588 (+415613us) service_pool.cc:225] Handling call
0426 13:21:16.936881 (+ 13293us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:16.937166 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:16.937205 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:16.937680 (+   475us) log.cc:842] Serialized 26 byte log entry
0426 13:21:17.371306 (+433626us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:17.371362 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.371449 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:17.943057 (+571608us) raft_consensus.cc:1671] finished
0426 13:21:17.943108 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:17.943433 (+   325us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":428262,"threads_started":1,"wal-append.queue_time_us":778663}
W20240426 13:21:18.290683 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.004s	sys 0.041s
I20240426 13:21:18.291630 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.286463 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:18.295101 25022 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.211s	user 0.013s	sys 0.025s
W20240426 13:21:18.299041 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.018s	sys 0.048s
I20240426 13:21:18.085836 30314 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.303192 25103 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:18.308974 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.326660 30484 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:18.323035 24950 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:18.287065 30379 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:18.299279 30404 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.023s	sys 0.036s
I20240426 13:21:18.301079 30473 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:18.296831 30189 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:17.732875 30154 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.595s
I20240426 13:21:18.330570 30416 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:18.335711 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.010s	sys 0.055s
I20240426 13:21:18.336117 30535 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.336216 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:18.336647 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2463) took 1331 ms. Trace:
W20240426 13:21:18.336911 30486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.027s	sys 0.072s
I20240426 13:21:18.337168 30409 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:18.338212 25104 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:18.338840 30443 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:18.338532 30193 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.331279 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.339897 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2254) took 2433 ms. Trace:
I20240426 13:21:18.292605 30073 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:18.350288 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.010s	sys 0.055s
I20240426 13:21:17.719734 30511 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.348992 25100 rpcz_store.cc:276] 0426 13:21:17.005703 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.304771 (+1299068us) service_pool.cc:225] Handling call
0426 13:21:18.335719 (+ 30948us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.336209 (+   490us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.336260 (+    51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.336331 (+    71us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.336362 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.336425 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.336567 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.351291 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 720) took 1333 ms. Trace:
I20240426 13:21:18.360348 25031 rpcz_store.cc:276] 0426 13:21:17.017718 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.298248 (+1280530us) service_pool.cc:225] Handling call
0426 13:21:18.351014 (+ 52766us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.339905 30217 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:18.357636 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.355463 30182 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.302083 25623 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:21:18.356348 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.374548 25097 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:18.374679 30372 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.375854 30423 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.377198 30501 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.377884 30372 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.377977 30501 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:18.378937 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 817679us
I20240426 13:21:18.380751 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:18.381021 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 410831us
I20240426 13:21:18.382324 30198 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:18.382962 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:18.383111 30198 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.383925 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2461) took 1378 ms. Trace:
I20240426 13:21:18.385347 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2880) took 1671 ms. Trace:
I20240426 13:21:18.369663 25025 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:18.386220 30466 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:18.386389 30464 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:18.386811 30466 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.386951 30464 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:18.387459 30464 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:18.356536 24953 rpcz_store.cc:276] 0426 13:21:15.906591 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:15.906705 (+   114us) service_pool.cc:225] Handling call
0426 13:21:15.906987 (+   282us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:15.907180 (+   193us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:15.907205 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:16.024616 (+117411us) log.cc:842] Serialized 26 byte log entry
0426 13:21:16.157560 (+132944us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:16.173196 (+ 15636us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:16.173293 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:17.790916 (+1617623us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:18.317795 (+526879us) raft_consensus.cc:1671] finished
0426 13:21:18.321491 (+  3696us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.339660 (+ 18169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122764032,"thread_start_us":41354,"threads_started":1,"wal-append.queue_time_us":326685}
I20240426 13:21:18.389928 30462 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:18.390019 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 732752us
I20240426 13:21:18.369738 30507 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.391551 30498 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:18.392467 30522 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:18.393005 30426 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.393191 30522 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.395735 30432 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.395932 30416 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.344198 30304 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.396795 30443 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.397408 30304 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.399174 30443 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.399597 30304 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.367626 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.400225 30379 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.400856 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2265) took 1729 ms. Trace:
I20240426 13:21:18.375402 30398 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:18.401288 30460 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:18.402047 30460 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.402711 30379 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.402813 30073 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.403208 30193 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.403843 30193 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:18.375782 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.377215 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2098) took 1152 ms. Trace:
I20240426 13:21:18.405133 24947 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:18.406307 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2114) took 1199 ms. Trace:
I20240426 13:21:18.406507 30406 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.331058 30281 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.406805 25024 rpcz_store.cc:276] 0426 13:21:17.206726 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.298570 (+1091844us) service_pool.cc:225] Handling call
0426 13:21:18.375618 (+ 77048us) spinlock_profiling.cc:243] Waited 903 us on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:18.406245 (+ 30627us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2076416}
I20240426 13:21:18.400768 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2119) took 1007 ms. Trace:
I20240426 13:21:18.387501 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.332844 30469 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:18.376852 30423 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.408488 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:18.370302 30486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.027s	sys 0.073s
I20240426 13:21:18.408624 25027 rpcz_store.cc:276] 0426 13:21:17.393167 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.329037 (+935870us) service_pool.cc:225] Handling call
0426 13:21:18.400687 (+ 71650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.411705 30193 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.413166 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2465) took 1399 ms. Trace:
I20240426 13:21:18.401311 24949 rpcz_store.cc:276] 0426 13:21:16.671797 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.672061 (+   264us) service_pool.cc:225] Handling call
0426 13:21:16.732731 (+ 60670us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.732942 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:21:17.034333 (+301391us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.400345 (+1366012us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.400439 (+    94us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:18.400561 (+   122us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.400663 (+   102us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.400786 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56898,"prepare.queue_time_us":427923,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":3281,"raft.queue_time_us":60233,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":182,"thread_start_us":1570498,"threads_started":2,"wal-append.queue_time_us":718978}
I20240426 13:21:18.413527 25102 rpcz_store.cc:276] 0426 13:21:17.013943 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.304907 (+1290964us) service_pool.cc:225] Handling call
0426 13:21:18.330852 (+ 25945us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.412820 (+ 81968us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.412852 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.412915 (+    63us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.412949 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.413000 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.413094 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.409550 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:18.409180 30469 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.416168 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.415139 25102 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:18.416240 30201 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.417277 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2100) took 1019 ms. Trace:
I20240426 13:21:18.356168 30433 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.417842 25102 rpcz_store.cc:276] 0426 13:21:17.397977 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.414520 (+1016543us) service_pool.cc:225] Handling call
0426 13:21:18.414898 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.417038 (+  2140us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.417210 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.406512 25105 rpcz_store.cc:276] 0426 13:21:17.224356 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.329210 (+1104854us) service_pool.cc:225] Handling call
0426 13:21:18.362552 (+ 33342us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.376852 (+ 14300us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.376883 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.376951 (+    68us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:18.376982 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.377037 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.377165 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.419409 30433 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.420203 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2101) took 1011 ms. Trace:
I20240426 13:21:18.420320 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:18.420619 25102 rpcz_store.cc:276] 0426 13:21:17.409124 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.419019 (+1009895us) service_pool.cc:225] Handling call
0426 13:21:18.419439 (+   420us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.419787 (+   348us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.419810 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.419878 (+    68us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.419914 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.419989 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.420115 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:18.421603 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68855.5us
I20240426 13:21:18.421974 30511 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.404932 30477 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.422539 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:18.417420 30469 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.422955 30477 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:18.423831 30477 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:18.411356 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.425249 25024 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:18.426404 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:18.408870 30281 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.378317 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2467) took 1339 ms. Trace:
I20240426 13:21:18.427372 25024 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:18.414425 30193 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.428557 25102 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:18.428695 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.429420 30191 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.429049 30217 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.427331 25104 rpcz_store.cc:276] 0426 13:21:17.038866 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.054487 (+ 15621us) service_pool.cc:225] Handling call
0426 13:21:18.378039 (+1323552us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.378237 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2896680832}
I20240426 13:21:18.384400 25096 rpcz_store.cc:276] 0426 13:21:17.005083 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.972502 (+967419us) service_pool.cc:225] Handling call
0426 13:21:17.973329 (+   827us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:17.974005 (+   676us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:17.974771 (+   766us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.383409 (+408638us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.383464 (+    55us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.383547 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.383612 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.383733 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105401,"prepare.queue_time_us":110671,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":7460,"thread_start_us":318097,"threads_started":1}
I20240426 13:21:18.393688 30426 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:18.373030 24951 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20240426 13:21:18.429028 25102 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:18.441385 30409 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.434842 30398 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:18.444861 30398 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:18.445230 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2780) took 1259 ms. Trace:
I20240426 13:21:18.445657 24951 rpcz_store.cc:276] 0426 13:21:17.185550 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.342260 (+1156710us) service_pool.cc:225] Handling call
0426 13:21:18.366849 (+ 24589us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.437426 (+ 70577us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.445158 (+  7732us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.446604 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:46260
I20240426 13:21:18.447178 24951 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:18.447527 24951 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:18.410547 30423 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.396054 30518 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:18.446677 30460 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.387378 30466 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:18.428804 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2093) took 1921 ms. Trace:
W20240426 13:21:18.367491 30448 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.035s	sys 0.068s
W20240426 13:21:18.477701 30448 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.035s	sys 0.069s
I20240426 13:21:18.478474 30448 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:18.385723 30198 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.479281 30448 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.464370 30409 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:18.464388 30521 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.007s	sys 0.020s
I20240426 13:21:18.465207 30426 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.466224 30182 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.366676 30484 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.467020 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2883) took 1737 ms. Trace:
I20240426 13:21:18.469928 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.458751 25099 rpcz_store.cc:276] 0426 13:21:16.506853 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.509037 (+  2184us) service_pool.cc:225] Handling call
0426 13:21:16.509305 (+   268us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:16.514167 (+  4862us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:16.514193 (+    26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:16.514636 (+   443us) log.cc:842] Serialized 52 byte log entry
0426 13:21:17.359930 (+845294us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:21:17.359986 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:17.360113 (+   127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:18.428316 (+1068203us) raft_consensus.cc:1671] finished
0426 13:21:18.428378 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.428644 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4548,"raft.queue_time_us":87938,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":7471,"thread_start_us":773679,"threads_started":1,"wal-append.queue_time_us":960867}
I20240426 13:21:18.404879 30410 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:18.470728 30516 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.000s	sys 0.020s
I20240426 13:21:18.470368 25102 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.476154 30507 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:18.476872 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2773) took 1588 ms (client timeout 1624 ms). Trace:
W20240426 13:21:18.430176 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 885971us
I20240426 13:21:18.391414 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:18.483242 30433 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:18.538254 24952 rpcz_store.cc:269] 0426 13:21:16.887888 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.911339 (+ 23451us) service_pool.cc:225] Handling call
0426 13:21:18.476534 (+1565195us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54002ca6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.476692 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760293248}
I20240426 13:21:18.484155 25028 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:18.485075 30473 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.487368 24951 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.489276 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2785) took 1302 ms. Trace:
I20240426 13:21:18.549461 25030 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:18.549592 24949 rpcz_store.cc:276] 0426 13:21:17.186854 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.414043 (+1227189us) service_pool.cc:225] Handling call
0426 13:21:18.414453 (+   410us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.488986 (+ 74533us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.489010 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.489064 (+    54us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.489094 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.489145 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.489231 (+    86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.489975 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.395726 30522 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.497258 30217 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.385775 25022 rpcz_store.cc:276] 0426 13:21:16.714236 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.934283 (+220047us) service_pool.cc:225] Handling call
0426 13:21:16.934549 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.976293 (+ 41744us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:16.999325 (+ 23032us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.384198 (+1384873us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.384302 (+   104us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.384405 (+   103us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.385095 (+   690us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:18.385131 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.385287 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23163,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":23720,"spinlock_wait_cycles":95556480,"thread_start_us":1307784,"threads_started":1,"wal-append.queue_time_us":755304}
I20240426 13:21:18.499531 30456 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:18.501837 30340 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.503561 30501 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:18.495862 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 936) took 2314 ms (client timeout 2216 ms). Trace:
I20240426 13:21:18.505496 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2786) took 1318 ms. Trace:
I20240426 13:21:18.505554 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:18.506595 30241 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:18.511039 30514 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.021s	sys 0.023s
I20240426 13:21:18.511590 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2278) took 1088 ms. Trace:
I20240426 13:21:18.567654 24947 rpcz_store.cc:276] 0426 13:21:17.423232 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.315796 (+892564us) service_pool.cc:225] Handling call
0426 13:21:18.511348 (+195552us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b54002cbfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.511493 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106066176}
I20240426 13:21:18.530659 25024 rpcz_store.cc:276] 0426 13:21:16.729553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.466198 (+1736645us) service_pool.cc:225] Handling call
0426 13:21:18.466542 (+   344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.466724 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.466748 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.466783 (+    35us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.466812 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.466864 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.466956 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.540094 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.541291 25029 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
W20240426 13:21:18.568857 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2889) took 1485 ms (client timeout 1657 ms). Trace:
I20240426 13:21:18.569007 29429 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:18.524616 30521 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.007s	sys 0.021s
I20240426 13:21:18.548077 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 937) took 2362 ms. Trace:
I20240426 13:21:18.540809 25623 raft_consensus.cc:595] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.570008 25619 rpcz_store.cc:276] 0426 13:21:16.185104 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.220187 (+ 35083us) service_pool.cc:225] Handling call
0426 13:21:16.220499 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:16.828153 (+607654us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:16.828194 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:16.828244 (+    50us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:16.828284 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.538101 (+1709817us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:18.547680 (+  9579us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.547879 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396893952}
W20240426 13:21:18.569290 25029 rpcz_store.cc:269] 0426 13:21:17.083669 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.946750 (+863081us) service_pool.cc:225] Handling call
0426 13:21:18.568803 (+622053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.571724 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:18.572388 25029 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:18.560391 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2117) took 1197 ms. Trace:
I20240426 13:21:18.555193 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 974) took 1456 ms. Trace:
I20240426 13:21:18.574018 24956 rpcz_store.cc:276] 0426 13:21:17.089455 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.344998 (+1255543us) service_pool.cc:225] Handling call
0426 13:21:18.510474 (+165476us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.510682 (+   208us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.510706 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.510744 (+    38us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.510780 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.510837 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.546003 (+ 35166us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b5400078fd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.546136 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117965184}
I20240426 13:21:18.564990 24951 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.575510 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:46260 (request call id 975) took 1375 ms. Trace:
I20240426 13:21:18.575891 24951 rpcz_store.cc:276] 0426 13:21:17.200195 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.446382 (+1246187us) service_pool.cc:225] Handling call
0426 13:21:18.575419 (+129037us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.576552 30433 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.578804 30433 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.579309 30317 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.579959 30317 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.582099 30317 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:18.588284 30315 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.510s	user 0.013s	sys 0.007s
I20240426 13:21:18.569207 25031 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:18.589432 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.590696 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.591696 25031 raft_consensus.cc:2396] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20240426 13:21:18.573135 25030 rpcz_store.cc:276] 0426 13:21:17.352839 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.303266 (+950427us) service_pool.cc:225] Handling call
0426 13:21:18.469808 (+166542us) spinlock_profiling.cc:243] Waited 7.05 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:18.549992 (+ 80184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16213760}
I20240426 13:21:18.573333 25619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:38842
I20240426 13:21:18.592515 25619 raft_consensus.cc:595] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.592957 25619 raft_consensus.cc:603] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:18.593200 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2887) took 1619 ms. Trace:
I20240426 13:21:18.594033 30484 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.594695 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:18.595232 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:18.595666 30314 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.595917 25024 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:18.597167 25024 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:18.597440 30426 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.597755 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 721) took 1579 ms. Trace:
W20240426 13:21:18.598474 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 207281us
W20240426 13:21:18.562455 25617 rpcz_store.cc:269] 0426 13:21:16.181028 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:16.181169 (+   141us) service_pool.cc:225] Handling call
0426 13:21:18.495519 (+2314350us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.495664 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5184033536}
I20240426 13:21:18.593549 25030 rpcz_store.cc:276] 0426 13:21:16.973906 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.592347 (+1618441us) service_pool.cc:225] Handling call
0426 13:21:18.592676 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.592857 (+   181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.592880 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.592927 (+    47us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:18.592985 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.593039 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.593135 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.562561 24950 rpcz_store.cc:276] 0426 13:21:17.187118 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.915258 (+728140us) service_pool.cc:225] Handling call
0426 13:21:18.505322 (+590064us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b5400078fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.505433 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90093824}
I20240426 13:21:18.545208 30535 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.598198 25024 rpcz_store.cc:276] 0426 13:21:17.018031 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.596611 (+1578580us) service_pool.cc:225] Handling call
0426 13:21:18.596993 (+   382us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.597518 (+   525us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.597684 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.603005 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:18.600739 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 724) took 1576 ms (client timeout 2002 ms). Trace:
W20240426 13:21:18.605526 25023 rpcz_store.cc:269] 0426 13:21:17.024142 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.298754 (+1274612us) service_pool.cc:225] Handling call
0426 13:21:18.489859 (+191105us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:18.600685 (+110826us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16982144}
W20240426 13:21:18.559504 30516 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.000s	sys 0.021s
I20240426 13:21:18.600382 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:18.608417 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102973us
I20240426 13:21:18.605104 25022 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
W20240426 13:21:18.602644 30514 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.021s	sys 0.024s
I20240426 13:21:18.513993 30189 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.627365 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:18.628265 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:18.633507 30473 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.634217 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2783) took 1413 ms. Trace:
I20240426 13:21:18.634675 24948 rpcz_store.cc:276] 0426 13:21:17.186312 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.353236 (+1166924us) service_pool.cc:225] Handling call
0426 13:21:18.514233 (+160997us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.550679 (+ 36446us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.550705 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.559656 (+  8951us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:18.599488 (+ 39832us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.599549 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.599651 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.612619 29429 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:18.637917 24949 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:18.646158 25029 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:18.652112 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 976) took 1427 ms. Trace:
I20240426 13:21:18.652534 24949 rpcz_store.cc:276] 0426 13:21:17.225168 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.599619 (+1374451us) service_pool.cc:225] Handling call
0426 13:21:18.623963 (+ 24344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.651770 (+ 27807us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.652028 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.658197 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2116) took 1384 ms. Trace:
I20240426 13:21:18.662003 25029 rpcz_store.cc:276] 0426 13:21:17.273478 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.657397 (+1383919us) service_pool.cc:225] Handling call
0426 13:21:18.657716 (+   319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.657897 (+   181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.657918 (+    21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.657969 (+    51us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.657997 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.658043 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.658127 (+    84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.663324 25029 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:18.663858 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2118) took 1294 ms. Trace:
I20240426 13:21:18.662003 25103 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:18.664780 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2109) took 1108 ms. Trace:
I20240426 13:21:18.665074 25103 rpcz_store.cc:276] 0426 13:21:17.556133 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:17.980455 (+424322us) service_pool.cc:225] Handling call
0426 13:21:18.664730 (+684275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.646015 29429 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.670092 29429 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:18.673295 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53364.7us
I20240426 13:21:18.661309 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2466) took 1637 ms. Trace:
I20240426 13:21:18.664162 25029 rpcz_store.cc:276] 0426 13:21:17.369349 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.662872 (+1293523us) service_pool.cc:225] Handling call
0426 13:21:18.663172 (+   300us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.663681 (+   509us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.663815 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.676916 25023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:21:18.677645 25022 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:18.677852 25023 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:18.678387 25023 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:18.678499 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2891) took 1546 ms. Trace:
I20240426 13:21:18.679085 25031 rpcz_store.cc:276] 0426 13:21:17.114884 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.617205 (+1502321us) service_pool.cc:225] Handling call
0426 13:21:18.649167 (+ 31962us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.661125 (+ 11958us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.661164 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.661219 (+    55us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.661262 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.661349 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.661461 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.689549 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2115) took 1478 ms. Trace:
I20240426 13:21:18.694628 25022 rpcz_store.cc:276] 0426 13:21:17.207313 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.618372 (+1411059us) service_pool.cc:225] Handling call
0426 13:21:18.654884 (+ 36512us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.678112 (+ 23228us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.685467 (+  7355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.674489 25101 rpcz_store.cc:276] 0426 13:21:17.023493 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.323834 (+1300341us) service_pool.cc:225] Handling call
0426 13:21:18.348502 (+ 24668us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.638251 (+289749us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.638290 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.638342 (+    52us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:18.638381 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.661055 (+ 22674us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:18.661113 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.661252 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4321,"spinlock_wait_cycles":607160064}
W20240426 13:21:18.702237 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 184967us
I20240426 13:21:18.620787 30241 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.693854 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:18.712566 25029 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20240426 13:21:18.716924 30535 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.720105 30410 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.720448 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.720633 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.721526 30475 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.723229 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.723434 30475 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.723865 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.724901 25025 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:18.725025 29429 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.725783 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2120) took 1308 ms. Trace:
I20240426 13:21:18.721028 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.714517 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2126) took 1155 ms. Trace:
I20240426 13:21:18.728325 25029 rpcz_store.cc:276] 0426 13:21:17.558873 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.679454 (+1120581us) service_pool.cc:225] Handling call
0426 13:21:18.708733 (+ 29279us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:18.714234 (+  5501us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.714457 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.729604 25026 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:18.726467 25025 rpcz_store.cc:276] 0426 13:21:17.417562 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.415816 (+998254us) service_pool.cc:225] Handling call
0426 13:21:18.725612 (+309796us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.725735 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707922304}
I20240426 13:21:18.725557 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2471) took 1362 ms. Trace:
I20240426 13:21:18.727432 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 722) took 1709 ms. Trace:
I20240426 13:21:18.730787 25022 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20240426 13:21:18.734292 25030 rpcz_store.cc:276] 0426 13:21:17.018321 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.612941 (+1594620us) service_pool.cc:225] Handling call
0426 13:21:18.661787 (+ 48846us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.709923 (+ 48136us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.709947 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.727132 (+ 17185us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:18.727185 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.727268 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.727378 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.734747 25022 pending_rounds.cc:76] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 8
I20240426 13:21:18.702092 30398 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.743048 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2884) took 1764 ms. Trace:
I20240426 13:21:18.743556 25026 rpcz_store.cc:276] 0426 13:21:16.973033 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.444637 (+1471604us) service_pool.cc:225] Handling call
0426 13:21:18.725745 (+281108us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.733411 (+  7666us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.736843 (+  3432us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400055fd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:18.737005 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":645506816}
I20240426 13:21:18.741161 25022 pending_rounds.cc:97] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20240426 13:21:18.747045 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2127) took 1183 ms. Trace:
I20240426 13:21:18.749099 25031 rpcz_store.cc:276] 0426 13:21:17.564023 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.699223 (+1135200us) service_pool.cc:225] Handling call
0426 13:21:18.734597 (+ 35374us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.734799 (+   202us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.734833 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.741586 (+  6753us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.746780 (+  5194us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.746857 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.746986 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.746481 25022 raft_consensus.cc:2866] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:18.753283 30559 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.064s	user 0.001s	sys 0.001s
I20240426 13:21:18.752529 29429 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.753991 25028 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:18.748452 30473 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:18.748221 30501 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:18.720692 30410 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.732846 30464 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.734849 25098 rpcz_store.cc:276] 0426 13:21:17.357110 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.371795 (+1014685us) service_pool.cc:225] Handling call
0426 13:21:18.416380 (+ 44585us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.416737 (+   357us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:18.417010 (+   273us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.718749 (+301739us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.718815 (+    66us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:18.718876 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.718962 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.719085 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3015,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":4294,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164,"thread_start_us":296776,"threads_started":1}
I20240426 13:21:18.773818 25102 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
W20240426 13:21:18.776877 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102831us
I20240426 13:21:18.780462 25030 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20240426 13:21:18.786309 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 403502us
W20240426 13:21:18.788345 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88040.9us
I20240426 13:21:18.768792 30475 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:18.789547 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:18.790405 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 384552us
I20240426 13:21:18.810668 30496 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:21:18.811272 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:18.811298 30496 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:18.815057 30496 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.816208 30189 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:18.816426 30564 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20240426 13:21:18.816519 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:18.840098 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2886) took 1866 ms. Trace:
I20240426 13:21:18.840536 25027 rpcz_store.cc:276] 0426 13:21:16.973641 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.447415 (+1473774us) service_pool.cc:225] Handling call
0426 13:21:18.447733 (+   318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.448068 (+   335us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:18.761889 (+313821us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.839772 (+ 77883us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.839821 (+    49us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:18.839866 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.839926 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.840026 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":69,"thread_start_us":373242,"threads_started":2}
I20240426 13:21:18.842576 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2781) took 1656 ms. Trace:
I20240426 13:21:18.842936 24954 rpcz_store.cc:276] 0426 13:21:17.185800 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.352813 (+1167013us) service_pool.cc:225] Handling call
0426 13:21:18.409413 (+ 56600us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.409801 (+   388us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:18.755906 (+346105us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.842275 (+ 86369us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.842320 (+    45us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.842362 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.842422 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.842516 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":417719,"raft.queue_time_us":83,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":403948,"threads_started":2}
I20240426 13:21:18.801237 30448 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:18.849673 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44068.6us
I20240426 13:21:18.850209 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:18.848120 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:18.853523 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2784) took 1667 ms. Trace:
I20240426 13:21:18.854905 30241 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.856462 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.858878 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.863646 24951 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
W20240426 13:21:18.865116 30563 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20240426 13:21:18.868548 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.874011 30410 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.863950 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.877504 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:18.861521 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2782) took 1675 ms. Trace:
W20240426 13:21:18.882258 30449 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20240426 13:21:18.883903 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2279) took 1421 ms. Trace:
I20240426 13:21:18.885324 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:18.888362 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:18.889473 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:18.885155 24952 rpcz_store.cc:276] 0426 13:21:17.462720 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.566012 (+1103292us) service_pool.cc:225] Handling call
0426 13:21:18.588977 (+ 22965us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:18.589180 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.589208 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:18.589698 (+   490us) log.cc:842] Serialized 26 byte log entry
0426 13:21:18.828547 (+238849us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:18.828583 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.828647 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:18.883696 (+ 55049us) raft_consensus.cc:1671] finished
0426 13:21:18.883714 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.883840 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":231009,"threads_started":1,"wal-append.queue_time_us":289735}
I20240426 13:21:18.892356 24954 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:18.887830 25023 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.848944 30475 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.901611 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:18.854024 24953 rpcz_store.cc:276] 0426 13:21:17.186578 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.389108 (+1202530us) service_pool.cc:225] Handling call
0426 13:21:18.389569 (+   461us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.389965 (+   396us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:18.700729 (+310764us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.853148 (+152419us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.853213 (+    65us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:18.853264 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.853340 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.853462 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":425039,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":3939,"spinlock_wait_cycles":14735616,"thread_start_us":420807,"threads_started":2,"wal-append.queue_time_us":170309}
I20240426 13:21:18.891312 30398 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.896451 30501 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:18.906244 25023 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.907419 29429 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:18.910307 30501 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:18.911485 30501 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:18.920485 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 97685.1us
I20240426 13:21:18.920614 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 723) took 1902 ms. Trace:
I20240426 13:21:18.921640 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:18.921216 25024 rpcz_store.cc:276] 0426 13:21:17.018607 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.606617 (+1588010us) service_pool.cc:225] Handling call
0426 13:21:18.647940 (+ 41323us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:18.662485 (+ 14545us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.662525 (+    40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:18.663275 (+   750us) log.cc:842] Serialized 52 byte log entry
0426 13:21:18.847379 (+184104us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.847455 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.847558 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:18.920295 (+ 72737us) raft_consensus.cc:1671] finished
0426 13:21:18.920327 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.920517 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3228,"raft.queue_time_us":21344,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":163453,"threads_started":1,"wal-append.queue_time_us":241899}
I20240426 13:21:18.923179 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:18.923620 30473 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:18.924098 30473 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:18.924285 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51366 (request call id 2888) took 1825 ms. Trace:
I20240426 13:21:18.924620 30473 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:18.924683 25023 rpcz_store.cc:276] 0426 13:21:17.083390 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.616913 (+1533523us) service_pool.cc:225] Handling call
0426 13:21:18.908622 (+291709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:18.924325 30464 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:18.924118 30315 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:18.928810 30475 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:18.933041 30201 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:18.933449 30201 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.933800 30201 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:18.936910 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:18.901391 30410 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:18.938105 30410 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:18.940517 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63167.2us
I20240426 13:21:18.941731 30315 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:18.943648 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:18.944945 30404 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:21:18.943877 30217 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:21:18.946328 30217 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:18.947454 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55674.8us
I20240426 13:21:18.947263 30217 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.949805 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.951102 30217 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:18.952759 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.953778 25623 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:18.953990 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.954787 25616 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:18.956537 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.957501 24952 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:18.957685 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.958600 24953 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:18.959182 24954 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:21:18.957746 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.960579 24948 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:18.961139 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.961637 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.962091 24952 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20240426 13:21:18.961547 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.962836 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:18.964566 24949 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:18.967276 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.970777 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:18.970992 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.971168 24952 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:21:18.972208 30507 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:18.972796 30448 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:18.973369 30448 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:18.974380 30448 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.976372 30535 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:18.976967 30535 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:18.978930 30535 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.973572 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:18.981765 24953 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:18.904853 30466 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:18.982697 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:18.984009 30466 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:18.984869 25100 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:18.986730 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2099) took 1591 ms. Trace:
I20240426 13:21:18.987187 25100 rpcz_store.cc:276] 0426 13:21:17.395219 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.371393 (+976174us) service_pool.cc:225] Handling call
0426 13:21:18.486394 (+115001us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.985377 (+498983us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.986453 (+  1076us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:18.986665 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1145801984}
I20240426 13:21:18.988317 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:18.991855 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 86443.1us
I20240426 13:21:18.965018 30410 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.018237 30294 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:19.016212 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:19.020895 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 123876us
I20240426 13:21:18.880908 24955 rpcz_store.cc:276] 0426 13:21:17.186057 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.352478 (+1166421us) service_pool.cc:225] Handling call
0426 13:21:18.387486 (+ 35008us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:18.387773 (+   287us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:18.737019 (+349246us) log.cc:842] Serialized 14 byte log entry
0426 13:21:18.860875 (+123856us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:18.860958 (+    83us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:18.861066 (+   108us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.861236 (+   170us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:18.861394 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":410427,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":18389,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87,"spinlock_wait_cycles":7115264,"thread_start_us":452772,"threads_started":2}
I20240426 13:21:19.029419 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.026407 29429 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:19.039973 30482 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:19.040416 29429 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.040320 24851 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 8 to 10, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 10 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:19.042971 29429 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:19.042721 24851 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:19.043622 29429 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:19.048830 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:19.052579 30535 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:19.054297 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.024427 30507 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:19.060655 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:19.059926 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
W20240426 13:21:19.071107 30578 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.065s	user 0.002s	sys 0.000s
I20240426 13:21:19.071954 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:19.077358 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49386.2us
I20240426 13:21:19.074666 24851 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:19.079057 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:19.080987 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 175727us
I20240426 13:21:19.088280 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2895) took 1149 ms. Trace:
I20240426 13:21:19.088728 25025 rpcz_store.cc:276] 0426 13:21:17.939349 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:18.733857 (+794508us) service_pool.cc:225] Handling call
0426 13:21:18.745886 (+ 12029us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:18.754124 (+  8238us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:18.754155 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:18.758134 (+  3979us) log.cc:842] Serialized 26 byte log entry
0426 13:21:18.990516 (+232382us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:18.990555 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:18.990621 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:19.088067 (+ 97446us) raft_consensus.cc:1671] finished
0426 13:21:19.088087 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:19.088215 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1923,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8252,"thread_start_us":232042,"threads_started":1,"wal-append.queue_time_us":328364}
I20240426 13:21:19.097797 30475 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:19.098301 30475 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:19.098706 30475 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:19.081486 29429 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:19.113227 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88036.5us
I20240426 13:21:19.112627 30294 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:19.124352 24851 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:19.129745 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.131502 25616 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:19.131350 25025 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:19.131899 25616 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:19.136166 30294 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.138707 30294 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:19.141072 30466 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:19.141598 30466 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:19.142033 30466 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.146503 25616 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.149919 29429 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.152788 25616 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:19.160079 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.160812 25023 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:19.160894 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.161780 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.162228 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.162416 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:19.162806 25022 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.163934 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.165325 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.166410 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.167157 25101 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:19.168156 25102 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20240426 13:21:19.169299 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.169421 25022 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:19.169575 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.170511 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.171471 25096 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:19.172389 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:19.173610 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.180326 24851 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 6 to 7, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 7 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:19.184194 30507 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.185119 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.183318 24851 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 9 to 10, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:19.193979 25616 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:19.203275 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.203888 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:21:19.205711 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 259891us
I20240426 13:21:19.205965 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:19.214942 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:19.220997 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.221557 25031 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:19.221500 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.224799 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.225656 25025 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:19.222347 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.227268 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.224799 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.230216 25096 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:19.232475 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.233701 30241 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:19.234544 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 166999us
I20240426 13:21:19.234319 30241 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.236332 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:19.236433 30241 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:19.236490 30201 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.238740 30201 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.239174 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.240105 25623 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:19.236507 30304 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:19.244719 30304 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:19.245781 30304 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:19.205698 30410 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20240426 13:21:19.247437 30304 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:19.248617 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.248898 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:19.221132 30191 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:19.265931 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.266888 24851 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 7 to 8, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 8 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:19.275130 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.262899 30217 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:19.278234 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:19.286084 25096 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20240426 13:21:19.268185 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4594251130000657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19310000000000002s for a response
I20240426 13:21:19.274487 30191 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:19.283007 30217 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:19.303699 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.284497 30404 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:19.307437 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.308862 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.309022 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.311004 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.314121 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.272817 30475 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.292399 30315 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:19.320761 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.320880 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:19.324252 25025 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:19.326372 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.328462 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.330956 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.312155 30217 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:19.333676 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.335004 30410 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:19.287909 30473 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.340953 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.316541 30404 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:19.347601 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.391741 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.395205 25619 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:19.394095 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:19.402606 25069 connection.cc:232] RPC call timeout handler was delayed by -1.3118690549999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
I20240426 13:21:19.403779 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.404409 30191 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.404914 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.406268 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.407580 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.408334 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.327190 30466 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.422055 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.427832 25102 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:19.351117 30404 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:19.423710 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.424340 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.424474 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.425100 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.426375 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.430935 25025 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 16. Current term is 17.
I20240426 13:21:19.431262 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:19.421762 30294 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:19.432075 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.404938 30475 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:19.433213 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.434513 30475 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:19.434960 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:19.436323 30440 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:19.436923 30440 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.439244 30440 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.440018 30253 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:19.440696 30253 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.443042 30253 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.447449 25096 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:21:19.448297 30475 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:19.448478 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.449133 30475 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:19.449491 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.449730 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.450690 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.452775 25099 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20240426 13:21:19.447856 25031 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:19.454782 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.455427 25102 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:19.455719 25102 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:19.456482 25102 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.457468 25102 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:19.459587 24850 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 274) took 39009 ms (client timeout 15000 ms). Trace:
I20240426 13:21:19.459884 30475 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.461367 30466 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.461993 30466 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:19.463354 30466 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:19.464426 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.465653 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.466202 25099 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.462319 30475 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.451974 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.470932 30587 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:19.471534 30587 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:19.471565 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.472671 30587 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.448566 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.474028 25097 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:19.474751 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.475582 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.480684 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.481738 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.482519 30201 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:19.482931 30201 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:19.483042 25097 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:19.483328 30201 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:19.486193 30217 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.487423 24953 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:19.488646 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.489486 30253 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:19.489869 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:19.490859 30217 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:19.491165 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.491991 30556 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:19.492241 30253 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:19.492430 30556 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:19.492837 30556 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:19.396131 30315 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.495167 30315 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.499815 30556 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.501117 25031 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:19.501226 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.501546 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.503036 30556 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:19.503674 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.506346 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.508018 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.508419 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.509364 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.509632 24954 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:21:19.509743 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:19.508656 24953 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:19.511023 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.514428 30501 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:19.514921 30501 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.516789 30501 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.520934 30473 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.521104 30404 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.522066 24851 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 7 to 8, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 8 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:19.434451 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.529301 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:19.431906 25624 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.523916 24851 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } }
I20240426 13:21:19.491477 30217 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:19.450737 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.538200 25099 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:19.537721 25102 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:19.540686 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.542096 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.543516 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:19.544785 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53725.9us
I20240426 13:21:19.546435 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.550769 24851 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:19.432816 30294 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:19.569620 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 483538us
I20240426 13:21:19.572815 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.577025 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.577811 25096 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:19.548213 30482 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.032s	sys 0.024s
I20240426 13:21:19.579110 25098 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20240426 13:21:19.588546 25617 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.003s	sys 0.041s
W20240426 13:21:19.589720 25617 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.004s	sys 0.041s
W20240426 13:21:19.600841 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 364548us
W20240426 13:21:19.584880 30482 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.032s	sys 0.025s
I20240426 13:21:19.621541 24851 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:19.638016 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.640651 29429 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:19.644765 30556 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:19.639822 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.586985 30404 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:19.653996 30585 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:21:19.653925 30475 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:19.657274 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:19.658053 25619 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:19.648667 24851 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 12 to 13, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 13 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:21:19.667392 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 436294us
I20240426 13:21:19.666603 25096 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:19.684162 30475 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.004s
I20240426 13:21:19.694341 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:19.673797 30217 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:19.700759 24948 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:19.702360 30294 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:19.532125 24850 rpcz_store.cc:269] 0426 13:20:40.449768 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:40.507824 (+ 58056us) service_pool.cc:225] Handling call
0426 13:21:19.459347 (+38951523us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:14.800128 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:14.840712 (+ 40584us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:14.840749 (+    37us) write_op.cc:427] Acquired schema lock
0426 13:21:14.840765 (+    16us) tablet.cc:660] Decoding operations
0426 13:21:14.841944 (+  1179us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:14.842019 (+    75us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:14.842051 (+    32us) tablet.cc:683] Acquiring locks for 14 operations
0426 13:21:14.843171 (+  1120us) tablet.cc:699] Row locks acquired
0426 13:21:14.843186 (+    15us) write_op.cc:260] PREPARE: finished
0426 13:21:14.843339 (+   153us) write_op.cc:270] Start()
0426 13:21:14.843431 (+    92us) write_op.cc:275] Timestamp: P: 1714137674843272 usec, L: 0
0426 13:21:14.843451 (+    20us) op_driver.cc:331] REPLICATION: starting
0426 13:21:14.851733 (+  8282us) log.cc:842] Serialized 5472 byte log entry
0426 13:21:16.077941 (+1226208us) op_driver.cc:444] REPLICATION: finished
0426 13:21:16.586927 (+508986us) write_op.cc:300] APPLY: starting
0426 13:21:16.597140 (+ 10213us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:17.597321 (+1000181us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:17.597451 (+   130us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:17.807492 (+210041us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:17.807762 (+   270us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0426 13:21:17.807806 (+    44us) write_op.cc:311] APPLY: finished
0426 13:21:17.808128 (+   322us) log.cc:842] Serialized 129 byte log entry
0426 13:21:18.681134 (+873006us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:18.751389 (+ 70255us) write_op.cc:446] Released schema lock
0426 13:21:18.751527 (+   138us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:18.817049 (+ 65522us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:18.817081 (+    32us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1319213,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":419493,"apply.run_cpu_time_us":45463,"apply.run_wall_time_us":2301129,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62675,"num_ops":14,"prepare.queue_time_us":470584,"prepare.run_cpu_time_us":23876,"prepare.run_wall_time_us":300934,"raft.queue_time_us":229689,"raft.run_cpu_time_us":5234,"raft.run_wall_time_us":256488,"replication_time_us":1197512,"thread_start_us":1530997,"threads_started":5,"wal-append.queue_time_us":1204533,"wal-append.run_cpu_time_us":8874,"wal-append.run_wall_time_us":1461612}]]}
W20240426 13:21:19.726161 30588 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20240426 13:21:19.725009 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.649539 30590 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20240426 13:21:19.693657 30449 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:19.737919 25624 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:19.740612 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:19.748602 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:19.703608 30217 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:19.763911 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:19.766438 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 334278us
I20240426 13:21:19.741413 30475 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:21:19.770845 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89396.8us
I20240426 13:21:19.739032 30449 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:19.751690 30304 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:19.786774 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 351893us
I20240426 13:21:19.761533 30217 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.807243 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.805354 25619 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:19.809552 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:19.818333 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.821483 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.823609 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.823891 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.826314 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.827489 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.830111 30294 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:19.832763 30294 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:19.833031 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.834185 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.832787 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:19.834281 30304 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:19.839314 30596 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:21:19.839131 30304 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.819425 30191 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:19.842521 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:19.843336 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:19.825803 30599 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:19.859318 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:21:19.861336 30304 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:19.862251 30482 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:19.862709 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.864387 30482 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:19.865034 30556 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:19.865201 30315 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:19.866945 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.833659 30294 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:19.865060 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:19.868954 30589 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20240426 13:21:19.870033 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.871522 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.872340 30566 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:19.876361 30241 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:19.876740 30241 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:19.877066 30241 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:19.877287 30507 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:19.877660 30507 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:19.877961 30507 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:19.864209 29429 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:19.841246 30590 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:19.875737 25022 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:19.884434 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.897781 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:19.854804 30191 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:19.865509 30522 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
W20240426 13:21:19.898739 25103 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.029s	sys 0.074s
W20240426 13:21:19.902714 25103 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.029s	sys 0.074s
I20240426 13:21:19.905460 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:19.906101 25031 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:19.898967 30591 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:19.948503 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.896059 30556 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:19.956110 30603 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:19.950433 30590 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:19.956912 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51396.1us
I20240426 13:21:19.975158 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:19.960260 25024 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:19.961207 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:19.963821 25027 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20240426 13:21:19.966605 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:19.930392 30253 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:19.973773 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.979600 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:19.978139 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 214315us
I20240426 13:21:19.985469 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:19.988525 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:20.000617 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:20.004516 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:19.974970 30501 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.002916 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:20.011590 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 207284us
W20240426 13:21:20.014292 24840 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 88603.8us
I20240426 13:21:19.981146 30315 raft_consensus.cc:2741] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Leader pre-election decision vote started in defunct term 15: won
I20240426 13:21:20.015681 24954 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:20.037904 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.038853 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.043022 25024 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:20.039814 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.045107 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:19.989761 30475 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:20.001749 30522 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.001272 30448 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.051115 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:20.050310 25031 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:20.056213 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.051487 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:20.055709 24952 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:19.931582 30449 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:19.979899 30556 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.049129 30340 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.068987 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:20.069789 24956 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:20.071147 30606 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:21:20.072261 30464 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:20.072706 30464 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:20.073558 30464 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.074806 30464 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
W20240426 13:21:20.079524 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 241626us
I20240426 13:21:20.053921 30501 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:19.990828 30590 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.089938 30556 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:20.097676 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:20.100534 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:19.990219 30603 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.065949 30522 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:20.060103 30294 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:20.083168 30501 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:20.115183 25589 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 16. Current term is 17.
I20240426 13:21:20.137576 25589 leader_election.cc:401] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:21:20.120383 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 119848us
I20240426 13:21:20.121475 24952 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20240426 13:21:20.085685 30616 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:20.110069 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53936.5us
I20240426 13:21:20.152974 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:20.052248 30191 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.162752 30616 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.163434 30616 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:20.168304 30610 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20240426 13:21:20.144954 30501 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:20.193011 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.146749 30522 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:20.196781 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 208311us
W20240426 13:21:20.207538 24948 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.506s	user 0.004s	sys 0.002s
I20240426 13:21:20.210285 24948 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:20.197794 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:20.215118 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65172.9us
W20240426 13:21:20.219746 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 111602us
I20240426 13:21:20.235446 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:20.235916 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88082.6us
I20240426 13:21:20.147444 30294 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.233801 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.218688 30599 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 17
I20240426 13:21:20.216902 30253 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:20.232435 30613 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20240426 13:21:20.151167 30475 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.217033 30507 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.245815 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.227226 30315 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20240426 13:21:20.242702 30340 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:20.244483 30616 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.252893 30535 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.271889 30409 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:20.275441 24954 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
W20240426 13:21:20.279106 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49560.1us
I20240426 13:21:20.272655 30448 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:20.259650 30591 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:20.304872 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 84309.5us
W20240426 13:21:20.305256 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48904.6us
I20240426 13:21:20.322839 30294 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:20.324122 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.299494 30566 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:20.330420 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114491us
I20240426 13:21:20.301294 30556 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:20.338402 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92660.8us
W20240426 13:21:20.374207 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 181282us
I20240426 13:21:20.374415 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:20.376626 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 178924us
I20240426 13:21:20.378434 25103 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240426 13:21:20.336556 30609 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20240426 13:21:20.390143 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.390866 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.388954 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:20.392514 30241 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.515s	user 0.005s	sys 0.005s
I20240426 13:21:20.392877 30241 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:20.393787 30627 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20240426 13:21:20.394107 30496 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:20.394790 30241 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:20.394578 30496 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:20.395478 30633 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:21:20.395587 24947 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20240426 13:21:20.396509 30496 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:20.401479 30639 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.078s	user 0.001s	sys 0.001s
W20240426 13:21:20.402637 30623 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.195s	user 0.002s	sys 0.000s
I20240426 13:21:20.421533 24995 leader_election.cc:398] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:20.428977 30410 raft_consensus.cc:979] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:20.375707 30590 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.389446 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.439893 24995 leader_election.cc:401] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20240426 13:21:20.403182 30603 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:20.316622 30253 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:20.457887 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70438.5us
I20240426 13:21:20.320009 30522 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.389863 30409 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:20.441931 30507 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:20.442534 30201 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.959s	user 0.009s	sys 0.003s
W20240426 13:21:20.447782 30217 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.000s
I20240426 13:21:20.477528 25025 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
W20240426 13:21:20.439949 30482 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.575s	user 0.007s	sys 0.000s
I20240426 13:21:20.480118 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:20.476708 30201 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.412485 30586 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.491717 30632 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.492148 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:20.491616 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 111302us
I20240426 13:21:20.493458 30217 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.505311 30315 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:20.505543 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.506512 30217 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:20.507237 30315 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:20.507642 30315 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:20.517823 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.492823 30482 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:20.492956 30604 raft_consensus.cc:1067] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
W20240426 13:21:20.564414 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52703.9us
W20240426 13:21:20.564778 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 161180us
I20240426 13:21:20.529575 30304 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:20.574163 30304 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:20.574580 30304 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:20.575146 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.576128 30645 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:20.576401 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.577798 30602 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.578370 30602 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:20.583010 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77490.8us
I20240426 13:21:20.583168 30304 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.585641 30304 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:20.585758 30625 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:20.586719 30602 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.588136 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.589432 30616 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:20.590193 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.591091 30632 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.591352 30616 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:20.591929 30616 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:20.592223 30628 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
W20240426 13:21:20.593533 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 216206us
W20240426 13:21:20.593043 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 271269us
I20240426 13:21:20.509702 30409 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:20.509372 30599 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 16. Current term is 17.
I20240426 13:21:20.583892 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.621508 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:20.622041 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:46260
I20240426 13:21:20.622671 24947 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:20.622890 30464 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:20.593374 30650 raft_consensus.cc:2741] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20240426 13:21:20.592161 30448 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:20.592758 30440 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.591717 30632 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.614720 30449 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:20.628921 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.617143 30593 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.617370 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:20.516356 30475 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14.
I20240426 13:21:20.631953 30650 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:20.632903 30650 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:20.633373 30650 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:20.632805 30593 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.634205 30650 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.635538 30448 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.623034 24947 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:20.533613 30501 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:20.625276 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106036us
I20240426 13:21:20.592476 30616 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:20.624585 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.616113 30473 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.636255 30201 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.623394 30464 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.612237 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:20.643770 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.632175 30632 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.668713 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.669394 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.654529 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.657374 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:20.659121 25623 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:20.659880 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:20.662808 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50676.2us
I20240426 13:21:20.669721 25097 pending_rounds.cc:76] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 9
I20240426 13:21:20.671610 25097 pending_rounds.cc:97] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20240426 13:21:20.624068 30217 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.671808 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.673648 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:20.675876 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:20.672166 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.671847 25097 raft_consensus.cc:2866] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:20.676870 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:20.685786 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.685547 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.699016 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:20.699522 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.701318 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.702160 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.699656 25026 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:20.635504 30593 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:20.706106 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48826.2us
I20240426 13:21:20.656636 30475 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.706610 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:20.725651 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.707340 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:20.714507 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:20.703917 25030 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:20.670454 30464 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:20.727326 25030 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:20.730305 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.731712 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.679452 30201 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:20.673203 30191 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.736231 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:20.735816 24953 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:21:20.700739 30650 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:20.684862 30556 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:20.704124 30217 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.753412 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.737870 30603 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.755473 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:20.704196 30602 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:20.724640 30641 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:20.670208 30625 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20240426 13:21:20.725244 30482 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:20.800155 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:20.662786 30473 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.729054 30340 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.509024 30315 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.774832 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20240426 13:21:20.777822 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92323.7us
I20240426 13:21:20.766810 25030 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.793313 25025 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:20.746685 30593 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:20.735188 30605 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.733798 30601 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:20.798308 30602 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:20.715706 30632 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20240426 13:21:20.737798 30651 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:20.799937 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:20.766395 30604 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:20.786696 30606 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240426 13:21:20.796352 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70736.6us
I20240426 13:21:20.777225 30611 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20240426 13:21:20.810477 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84065.7us
I20240426 13:21:20.819579 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.847201 25026 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:20.822667 25621 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:20.771647 30522 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20240426 13:21:20.823602 30660 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:20.826717 30631 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:20.828995 24947 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.794963 30466 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.852409 30654 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.852123 30631 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.852964 30473 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.854372 30631 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.791258 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 947) took 1016 ms. Trace:
I20240426 13:21:20.828943 25030 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:20.766073 30475 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20240426 13:21:20.797515 30641 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:20.819789 30603 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:20.731429 30409 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.822454 30630 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:20.817457 30566 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:20.859305 25099 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:20.820523 30593 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:20.854921 25096 rpcz_store.cc:276] 0426 13:21:19.775171 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:19.775321 (+   150us) service_pool.cc:225] Handling call
0426 13:21:19.809050 (+ 33729us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:19.817345 (+  8295us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:19.971066 (+153721us) log.cc:842] Serialized 14 byte log entry
0426 13:21:20.775709 (+804643us) log.cc:842] Serialized 14 byte log entry
0426 13:21:20.790031 (+ 14322us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:20.790172 (+   141us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:20.790316 (+   144us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:20.790735 (+   419us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:20.791031 (+   296us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":226063,"prepare.queue_time_us":150420,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":283623,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":157,"thread_start_us":683098,"threads_started":3,"wal-append.queue_time_us":495371}
I20240426 13:21:20.867614 24947 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:20.845185 30651 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:20.873068 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 118206us
I20240426 13:21:20.873503 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:20.875510 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 119953us
I20240426 13:21:20.869307 30604 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:20.911475 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.852433 30605 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20240426 13:21:20.868810 30501 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:20.918903 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.913940 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.870503 30593 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:20.915444 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:20.917577 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44150.3us
I20240426 13:21:20.920397 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:20.921222 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.921449 25024 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:20.921625 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:20.928436 25027 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20240426 13:21:20.930447 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 224963us
I20240426 13:21:20.935818 24947 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:20.936722 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.912798 30599 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:20.939971 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.942945 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:20.942945 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:20.943769 25624 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:20.943796 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:20.944250 25624 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:20.874723 30423 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20240426 13:21:20.945405 25621 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:20.871740 30603 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.945834 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:20.871251 30566 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.880010 30340 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.062s
I20240426 13:21:20.949739 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.955636 25096 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:20.880517 30630 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.949512 25624 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.973614 25624 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:20.973950 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:20.975718 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:20.982787 25027 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:20.986318 25100 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:20.987828 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.952746 30253 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:20.989432 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:20.989440 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:20.995064 25026 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:20.991089 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.957720 30340 consensus_queue.cc:1225] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:21:20.948999 30668 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:20.994355 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:20.944981 29429 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20240426 13:21:20.943048 30501 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20240426 13:21:20.980965 30507 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:20.945161 30599 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:20.937980 30641 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:20.956894 30651 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:20.993577 30649 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20240426 13:21:21.003624 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:21.001632 25024 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20240426 13:21:21.030159 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:21.035061 25624 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:21.002478 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82150.8us
I20240426 13:21:21.020535 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:21.003913 25027 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20240426 13:21:21.020949 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:21.085359 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:21.003366 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:21.002676 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95049.8us
W20240426 13:21:21.100713 30669 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20240426 13:21:20.996847 30625 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:21.091066 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:21.096518 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 110408us
I20240426 13:21:21.114192 25097 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:21.097592 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:21.118722 25096 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20240426 13:21:21.118913 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:21.089444 25026 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:21.094290 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:21.124688 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:21.088542 30661 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20240426 13:21:21.026326 30668 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:21.085578 30501 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:21.098416 25104 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:21.155270 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 159994us
I20240426 13:21:21.158944 25026 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:21.159371 25026 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:21.121088 30658 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.155s	user 0.001s	sys 0.003s
I20240426 13:21:21.160748 25105 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:21.170360 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 220569us
I20240426 13:21:21.090857 30641 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:21.093241 30473 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:21.086392 30507 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:21.088282 30566 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:21.087491 30630 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:21.175340 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:21.175621 25027 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:21.117830 30475 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.058s
W20240426 13:21:21.177273 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79774us
I20240426 13:21:20.999380 30659 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.160s
I20240426 13:21:21.149569 30404 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
W20240426 13:21:21.196015 30677 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:21:21.052301 30678 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.146s
I20240426 13:21:21.176517 25026 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:21.196920 25026 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:20.998905 30253 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:20.976171 30586 consensus_queue.cc:1159] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 14 status { last_received { term: 13 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:21.156098 30668 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:21.036986 30670 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.148s
I20240426 13:21:21.189023 25028 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:21.190816 30566 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:21.192390 30507 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:21.164137 30501 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:21.207417 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20240426 13:21:21.196189 30404 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:21.196923 30678 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20240426 13:21:21.180728 30473 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:21.237286 25616 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
W20240426 13:21:21.407233 30673 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.422s	user 0.001s	sys 0.000s
W20240426 13:21:21.862905 30616 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.193s	user 0.018s	sys 0.003s
I20240426 13:21:21.333606 25617 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:21.863322 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 953) took 1158 ms. Trace:
W20240426 13:21:21.296109 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 140394us
I20240426 13:21:21.289592 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:21.420912 25622 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:21.358507 25620 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:21.177980 30680 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:21.701856 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.041s	sys 0.128s
I20240426 13:21:21.356369 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:21.944778 25619 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:21.356251 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:21.363747 25026 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:21.946067 25098 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:21.468847 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1442) took 1089 ms. Trace:
W20240426 13:21:21.885383 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.014s	sys 0.001s
I20240426 13:21:21.932420 25102 rpcz_store.cc:276] 0426 13:21:20.704446 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:20.706953 (+  2507us) service_pool.cc:225] Handling call
0426 13:21:20.734913 (+ 27960us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:20.735130 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:20.735170 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:20.749132 (+ 13962us) log.cc:842] Serialized 26 byte log entry
0426 13:21:20.871189 (+122057us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:20.871224 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:20.871290 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:21.863123 (+991833us) raft_consensus.cc:1671] finished
0426 13:21:21.863143 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:21.863268 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":52868,"threads_started":1,"wal-append.queue_time_us":158223}
W20240426 13:21:21.947183 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.016s	sys 0.002s
W20240426 13:21:21.943048 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.044s	sys 0.130s
W20240426 13:21:21.948441 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 789703us
W20240426 13:21:21.483546 30464 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.737s	user 0.009s	sys 0.001s
I20240426 13:21:21.289659 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11fad756437243d2959f5998d6bf8309"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:21.953969 25101 raft_consensus.cc:595] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:21.237365 30664 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20240426 13:21:21.954393 25101 raft_consensus.cc:603] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:21.237983 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:21.833184 30593 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.896s	user 0.007s	sys 0.004s
I20240426 13:21:21.210024 30603 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:21.345371 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 225193us
I20240426 13:21:21.863380 30616 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:21.473662 30449 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.837s	user 0.008s	sys 0.002s
W20240426 13:21:21.778209 30602 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.930s	user 0.010s	sys 0.000s
W20240426 13:21:21.465597 30656 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.621s	user 0.003s	sys 0.000s
W20240426 13:21:21.579720 30650 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.757s	user 0.010s	sys 0.000s
I20240426 13:21:21.947036 25103 rpcz_store.cc:276] 0426 13:21:20.379085 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:20.379949 (+   864us) service_pool.cc:225] Handling call
0426 13:21:21.468189 (+1088240us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:21.468745 (+   556us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919489920}
W20240426 13:21:21.962909 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 673354us
I20240426 13:21:21.999814 30676 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.000381 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.000319 30676 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.001013 30602 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.001608 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.001809 25024 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:22.001757 30602 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:22.002470 30602 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.003283 30680 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:21.893685 30667 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:21.808688 30648 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:21.207244 30668 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:22.004139 30648 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:21.983232 25104 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:22.004148 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:21.844348 30641 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.665s	user 0.005s	sys 0.005s
I20240426 13:21:21.433316 30604 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:22.024945 30604 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:22.026266 30604 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:21.291353 30566 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:21.618060 30632 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:21.986093 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:21.949280 30681 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:21.963307 30672 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20240426 13:21:22.002761 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.028412 30632 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:21.438030 30599 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.030002 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.030872 30632 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.031163 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.031687 30599 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.032269 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.031772 30566 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.032631 30599 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.033427 30680 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.034032 30680 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.034519 30680 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.035027 30680 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:21.466800 30591 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:22.006086 25023 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:21.422221 30642 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:21.333151 25623 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20240426 13:21:21.982172 30665 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:22.054015 25027 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
W20240426 13:21:22.054527 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68648.2us
I20240426 13:21:22.075892 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:21.237663 30675 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.803s
W20240426 13:21:22.110535 30664 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20240426 13:21:22.128437 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.128482 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:21.361444 30501 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.003538 30602 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:22.045261 30681 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.046768 30648 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:21.963963 30464 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.113659 30642 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:22.115445 30253 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:22.202325 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 170118us
I20240426 13:21:22.220778 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:21.984756 30593 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.035861 30591 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:22.115964 30654 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.116199 30678 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:22.187402 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59007.8us
I20240426 13:21:22.035034 30599 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.209846 25621 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:22.235953 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 734) took 1030 ms. Trace:
I20240426 13:21:22.046770 30616 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:22.224359 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 930763us
I20240426 13:21:22.004391 30667 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.189466 30641 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.187047 30692 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.236492 25621 rpcz_store.cc:276] 0426 13:21:21.205116 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.205240 (+   124us) service_pool.cc:225] Handling call
0426 13:21:22.235884 (+1030644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:22.242655 25099 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:22.129972 30665 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.256057 30464 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.220592 30632 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.267926 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.273399 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.273676 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.274281 25028 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:22.277032 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.277129 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:22.183434 30404 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.278409 25031 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:21:22.283351 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.283628 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.236025 30648 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.207541 30681 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:22.271517 30692 raft_consensus.cc:465] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Not starting pre-election -- already a leader
I20240426 13:21:22.306246 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.306042 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.306422 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:22.295437 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.305578 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.304666 30440 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.305372 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.321739 25617 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:22.322651 25022 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:21:22.273103 30680 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:22.305264 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.314904 24952 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20240426 13:21:22.318301 24947 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:21:22.292209 25098 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:22.292001 25100 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:22.325616 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:22.295665 30694 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:22.326120 24956 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20240426 13:21:22.331667 24956 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 6
I20240426 13:21:22.259289 30678 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20240426 13:21:22.332006 24956 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20240426 13:21:22.332407 24956 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:22.303135 30683 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:22.237042 30654 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:22.337028 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 123720us
I20240426 13:21:22.338840 24951 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:21:22.305239 30665 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.277690 30632 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:22.328298 30692 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:22.258812 30702 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.092s
I20240426 13:21:22.348261 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.306924 30651 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20240426 13:21:22.329485 30473 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.356246 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.358579 24956 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:22.358839 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.359577 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.359953 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:22.360697 24954 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20240426 13:21:22.306722 30704 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20240426 13:21:22.357059 30692 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.334354 30440 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.340545 30683 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:22.290148 30684 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
W20240426 13:21:22.356215 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 135327us
I20240426 13:21:22.368046 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.368206 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.368669 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:22.346436 30689 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.370064 24951 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240426 13:21:22.374516 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.295181 30648 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.375123 24954 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:22.354781 30707 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.376780 30707 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.378947 30707 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.357216 30464 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.357336 30706 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.364450 30644 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20240426 13:21:22.363976 30641 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:22.390519 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2520) took 1211 ms. Trace:
I20240426 13:21:22.391602 30440 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.367952 30678 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.270964 30642 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.364619 30705 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:22.382098 30689 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.398639 30641 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.399034 30681 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:22.396449 25105 rpcz_store.cc:276] 0426 13:21:21.178682 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.178846 (+   164us) service_pool.cc:225] Handling call
0426 13:21:21.199076 (+ 20230us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:21.871454 (+672378us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:21.871506 (+    52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:21.889420 (+ 17914us) log.cc:842] Serialized 26 byte log entry
0426 13:21:22.194292 (+304872us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:21:22.194326 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.194409 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:22.390322 (+195913us) raft_consensus.cc:1671] finished
0426 13:21:22.390341 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:22.390466 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98823,"raft.queue_time_us":666253,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"thread_start_us":276019,"threads_started":1,"wal-append.queue_time_us":382906}
I20240426 13:21:22.410653 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2522) took 1231 ms. Trace:
I20240426 13:21:22.415567 24954 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 8
I20240426 13:21:22.381361 30717 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.416299 24954 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 16.9
W20240426 13:21:22.417531 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52589.7us
I20240426 13:21:22.418186 24954 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:22.421998 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:22.422490 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:22.424904 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.408948 30599 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20240426 13:21:22.426139 30702 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:22.426772 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 955) took 1712 ms. Trace:
I20240426 13:21:22.427045 30632 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.427096 25623 rpcz_store.cc:276] 0426 13:21:20.714855 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:20.715769 (+   914us) service_pool.cc:225] Handling call
0426 13:21:20.734531 (+ 18762us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:22.074771 (+1340240us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.426546 (+351775us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:22.426703 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1375859072}
W20240426 13:21:22.427749 30703 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:21:22.422432 30704 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.429198 30704 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.429606 30689 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.413271 30678 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.431105 30704 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:22.432005 30700 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.482s	user 0.000s	sys 0.001s
I20240426 13:21:22.431870 30678 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.423708 30717 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.434547 30448 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:22.434929 30448 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:22.434993 30717 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.435914 30448 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.438531 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2944) took 1242 ms. Trace:
I20240426 13:21:22.438627 30448 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:22.438936 25027 rpcz_store.cc:276] 0426 13:21:21.195902 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.206967 (+ 11065us) service_pool.cc:225] Handling call
0426 13:21:21.299417 (+ 92450us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.130166 (+830749us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.438323 (+308157us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54002add54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:22.438481 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1718920576}
I20240426 13:21:22.437705 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2156) took 1496 ms. Trace:
I20240426 13:21:22.440197 25024 rpcz_store.cc:276] 0426 13:21:20.941522 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.102819 (+161297us) service_pool.cc:225] Handling call
0426 13:21:21.113392 (+ 10573us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.002350 (+888958us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.437514 (+435164us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:22.437653 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2042660608}
I20240426 13:21:22.441226 30654 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.442219 30648 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.443315 30683 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.445207 30614 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:22.419739 30566 consensus_queue.cc:1159] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 15 status { last_received { term: 14 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:22.445850 30614 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:22.445358 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77220.5us
I20240426 13:21:22.407907 30630 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:22.400228 30593 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.449642 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.447773 30614 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.385715 30464 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.445255 30680 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.456310 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.456534 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.457566 30704 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:22.423190 30651 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.421299 30642 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:22.422279 30721 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:22.481353 30721 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.469816 30496 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:22.486261 30315 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:22.488602 30604 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:22.488775 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2517) took 1546 ms. Trace:
I20240426 13:21:22.489343 25099 rpcz_store.cc:276] 0426 13:21:20.946300 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:20.952142 (+  5842us) service_pool.cc:225] Handling call
0426 13:21:20.952360 (+   218us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.254868 (+1302508us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.488482 (+233614us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:22.488703 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2966835584}
I20240426 13:21:22.490582 30473 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.470800 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.491814 30692 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.492074 30680 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:22.492574 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.493165 30680 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:22.493496 30680 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:22.451939 30630 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:22.494302 30680 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.472681 30717 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20240426 13:21:22.489372 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.496603 30449 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:22.496860 30701 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20240426 13:21:22.461076 30667 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.471396 30704 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.500277 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:22.506525 25099 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.479045 30629 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:22.459138 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.497107 25022 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20240426 13:21:22.475010 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.499327 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.508173 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.504602 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:22.508658 30704 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:22.504725 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.511461 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:22.506846 25099 raft_consensus.cc:603] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:22.507627 24952 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:22.500036 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.513891 25028 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:22.505486 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.514400 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.506465 30423 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.516742 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.518493 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.495501 30630 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:22.520753 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 102848us
W20240426 13:21:22.526751 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66865.4us
I20240426 13:21:22.527612 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.530915 25619 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:22.529644 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:22.507237 30721 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.506217 30496 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.508898 30670 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.508929 30675 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.509088 30629 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:22.511808 30641 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.534554 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.528024 30680 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:22.516364 30720 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.534847 30201 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.549006 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.544229 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.497426 30651 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.552609 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.557303 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.542752 30482 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:22.559778 24949 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20240426 13:21:22.557888 30637 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:22.561461 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.561722 30632 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.563223 25622 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:21:22.563810 30676 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.564074 30440 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.568949 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.570650 25617 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20240426 13:21:22.546913 30665 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.572440 30629 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.548851 30681 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.574080 30482 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:22.574537 30482 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:22.577096 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.574586 30630 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:22.530508 30423 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.582901 24952 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:22.583250 24949 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 11. Current term is 12. Ops: []
I20240426 13:21:22.585394 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.537801 30667 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20240426 13:21:22.587849 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.588477 24956 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:22.589327 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:22.589993 24956 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:22.590669 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.591665 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.592499 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.593030 24956 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20240426 13:21:22.598584 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.541059 30496 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.604365 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.603246 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.602608 25621 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:22.605820 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:22.543751 30670 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.546711 30705 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.023s
I20240426 13:21:22.610121 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.545089 30644 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20240426 13:21:22.574630 30464 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.563036 30666 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:21:22.613662 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:22.593487 30681 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.615005 30593 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240426 13:21:22.617421 30732 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.618927 30496 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.619419 30423 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.620847 30716 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:22.621196 30718 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:22.622102 30651 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:22.624389 30723 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20240426 13:21:22.544667 30707 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20240426 13:21:22.628904 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.593541 30632 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:22.607858 30614 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.614764 30684 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:22.612912 30566 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.593308 30601 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20240426 13:21:22.655022 30716 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:22.633241 30648 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.549890 30675 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.608456 30667 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20240426 13:21:22.645246 30665 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:22.658146 30614 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.666373 30714 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20240426 13:21:22.694757 30614 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.695972 30670 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.688163 30648 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.663895 30716 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.697353 30716 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:22.698596 30648 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.636106 30681 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.702688 30681 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.636510 30672 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.556908 30720 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.707774 30720 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:22.709578 30722 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.281s	user 0.003s	sys 0.000s
I20240426 13:21:22.706061 30672 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.711401 30672 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.415720 25096 rpcz_store.cc:276] 0426 13:21:21.180111 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.180244 (+   133us) service_pool.cc:225] Handling call
0426 13:21:21.203966 (+ 23722us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:21.204900 (+   934us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:22.337568 (+1132668us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b180057dfc0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f
0426 13:21:22.337692 (+   124us) log.cc:842] Serialized 14 byte log entry
0426 13:21:22.410336 (+ 72644us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:22.410386 (+    50us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:22.410432 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.410487 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:22.410603 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122964,"prepare.queue_time_us":232510,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":39,"raft.queue_time_us":36400,"spinlock_wait_cycles":12476288,"thread_start_us":297910,"threads_started":2,"wal-append.queue_time_us":60738}
I20240426 13:21:22.713783 30689 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:22.714344 30689 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.716398 30689 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:22.717972 30734 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20240426 13:21:22.719233 30689 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.719709 30689 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.721679 30689 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.664299 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.647300 30718 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:22.645912 30732 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.684669 30566 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:22.653715 30632 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.723960 30732 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.724115 30632 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.612247 30602 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:22.725812 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61235.1us
I20240426 13:21:22.726019 30602 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.633365 30625 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20240426 13:21:22.726755 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.721937 30716 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.727420 30625 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:22.688472 30712 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.690021 30675 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.657799 30660 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:22.692178 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:22.669039 30668 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.665s	user 0.004s	sys 0.005s
I20240426 13:21:22.739276 25624 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:22.722360 30737 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20240426 13:21:22.634435 30704 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.626658 30482 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:22.741537 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74971.6us
I20240426 13:21:22.686414 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.728688 30716 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.753929 30665 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:22.692174 30688 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.729496 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:22.685734 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 299945us
W20240426 13:21:22.740904 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54553.3us
I20240426 13:21:22.748562 30637 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20240426 13:21:22.759203 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:22.749634 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98469.5us
I20240426 13:21:22.742514 30660 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:22.746346 30625 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:22.758054 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.762717 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:22.761626 24956 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:22.746490 30672 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.753252 30668 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.728291 30712 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.769780 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.776376 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.787523 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.806313 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.744413 30201 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20240426 13:21:22.816372 30712 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:22.816748 25096 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20240426 13:21:22.817111 25096 pending_rounds.cc:76] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 10
I20240426 13:21:22.817308 25096 pending_rounds.cc:97] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20240426 13:21:22.817523 25096 raft_consensus.cc:2866] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:22.818609 25024 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:22.818761 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.819438 25025 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:22.820600 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.809443 25028 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:22.823825 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.824086 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.810881 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:22.826496 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:22.821415 25031 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:22.827656 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:22.829563 30448 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:22.829941 30448 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:22.830135 25029 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:22.830271 30448 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:22.831001 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.832302 30632 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:22.833974 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.814137 30602 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:22.834565 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.834766 30691 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:22.835022 30602 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:22.835273 30691 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:22.835476 30602 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:22.835661 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.836220 30691 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.836803 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.837453 25022 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:22.837668 30482 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:22.838243 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.838693 30670 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:22.839046 30670 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:22.839190 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.839406 30670 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:22.841320 30501 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:22.782044 30440 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:22.843932 30440 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.844267 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.845440 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.756860 30688 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.846407 24949 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20240426 13:21:22.846817 24949 pending_rounds.cc:76] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 8
I20240426 13:21:22.847034 24949 pending_rounds.cc:97] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20240426 13:21:22.847295 24949 raft_consensus.cc:2866] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:22.851095 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.851675 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:22.852386 30704 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.853179 30739 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:22.853711 30739 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:22.854122 30739 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:22.856750 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.839252 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.860278 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.767073 30660 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:22.762670 30689 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:22.782264 30631 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:22.867839 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:22.878187 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:22.868244 25029 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:22.880296 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.883203 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:22.882184 30409 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:22.866667 30681 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:22.869176 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:22.867004 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2519) took 1700 ms. Trace:
I20240426 13:21:22.844312 30440 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:22.866582 30654 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:22.870241 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:22.832760 30735 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20240426 13:21:22.870836 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:22.871577 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.862614 25030 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:22.873718 25022 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:22.862154 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51357.4us
I20240426 13:21:22.878206 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.884351 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.765174 30625 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:22.862308 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:22.889005 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:22.882349 30404 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:22.895462 30409 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.810338 30672 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.992473 25104 rpcz_store.cc:276] 0426 13:21:21.166232 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:21.166371 (+   139us) service_pool.cc:225] Handling call
0426 13:21:21.167956 (+  1585us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.005586 (+837630us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:22.865051 (+859465us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54003e17d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:22.866953 (+  1902us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1873935104}
I20240426 13:21:22.895982 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:22.991521 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105948us
I20240426 13:21:22.783423 30665 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:22.998662 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.994362 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.995093 25105 raft_consensus.cc:595] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.000561 25105 raft_consensus.cc:603] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:23.001202 25096 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:23.001937 25103 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:23.002599 25029 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:22.995486 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:22.896654 30684 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:23.009307 30684 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:22.991379 25031 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:23.001494 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.005779 25097 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:23.016980 25022 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:23.018064 25022 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20240426 13:21:23.006146 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:23.024363 25099 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 12: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:22.896493 30648 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
I20240426 13:21:23.025775 30404 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:22.898847 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:23.027506 24949 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:23.027737 30404 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.031560 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7408112bace482693c999f678aa9d73"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:23.019024 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:23.018970 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:22.990681 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:22.963152 29429 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:23.034279 29429 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.036221 29429 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.037525 30522 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:23.033838 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 171624us
I20240426 13:21:23.041885 30522 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:23.042917 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.043144 30522 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:23.043526 30522 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:23.044313 30522 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.045611 30522 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:23.046131 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:23.043490 30684 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.047142 25104 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:23.047711 25101 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:23.024919 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:23.048530 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.025158 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.024938 30668 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.023653 25022 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:23.050246 25096 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:22.879065 30712 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.025970 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:23.051784 30712 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:22.929605 30736 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20240426 13:21:22.882603 30704 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.146s
I20240426 13:21:23.029074 30448 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.055613 25102 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [13.7-16.8]
I20240426 13:21:23.055812 30632 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:22.994500 30603 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:23.056854 30603 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.026003 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.057215 30448 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.008653 30672 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.057930 30672 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:23.057976 30712 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.058305 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:23.058796 30603 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:23.059787 30672 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:23.060550 30712 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:23.048502 30684 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:22.963611 30707 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.063980 30707 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.065915 30707 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:23.066545 25105 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20240426 13:21:23.067348 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.069006 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.069132 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:23.031380 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:22.995903 30625 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:23.083249 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.071818 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.088923 24948 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:23.089653 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.090366 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.091070 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:23.056982 30522 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:23.084005 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.085947 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:23.086175 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54883.2us
I20240426 13:21:23.074025 25101 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:23.079797 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.087957 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:23.077769 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:23.099048 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:23.099280 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:23.019045 30409 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.110142 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.097322 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.103942 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:23.112633 25617 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:23.104441 30681 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:23.105154 30604 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:23.071144 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:23.096992 30737 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:22.883103 30631 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.114022 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.156960 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:23.099197 30603 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:23.116964 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.107477 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.096901 30522 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:23.119166 30593 raft_consensus.cc:979] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:21:23.163081 25099 raft_consensus.cc:595] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.121438 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.123742 24955 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20240426 13:21:23.125239 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:23.154665 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.147639 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.157017 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:23.157827 24949 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:23.114768 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:23.166339 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:23.166464 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51742.9us
I20240426 13:21:23.167582 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.168208 30666 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:23.168617 30666 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:23.168964 30666 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:23.170590 30721 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:23.114996 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:23.170065 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.173059 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:23.173734 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:23.102267 30201 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20240426 13:21:23.173066 24949 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:23.175565 30671 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:23.175977 30671 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:23.176533 30660 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.176856 30671 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.177556 30689 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:23.177917 30671 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:23.178453 30660 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.177995 30689 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:23.179262 30689 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:23.122555 30632 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:23.180909 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:23.181660 30705 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:23.182041 30705 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:23.182410 30705 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:23.099299 30404 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:23.183890 30404 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.184283 30404 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:23.175864 30593 raft_consensus.cc:1067] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:21:23.190483 30688 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.192231 30742 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:23.192965 30201 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:23.079497 30739 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.180960 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.197901 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:23.193773 25105 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:23.164927 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:23.195047 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87624.1us
I20240426 13:21:23.160120 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.105823 30496 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:23.152854 30409 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:23.180830 30632 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:23.203478 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.204702 25096 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:23.175542 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.114351 30710 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:23.199544 25619 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:23.211781 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:23.159569 30616 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:23.216920 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.216835 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:23.213179 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.218624 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:23.221276 25027 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:23.220360 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.213318 25617 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:23.119531 30707 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20240426 13:21:23.164312 30522 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:23.200857 30614 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:23.197382 30201 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:23.201310 30678 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:23.141057 30440 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.203048 30409 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.175868 30670 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.306970 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:23.394997 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 181868us
W20240426 13:21:23.397545 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276181us
I20240426 13:21:23.406566 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.407994 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.211057 30632 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:23.396145 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:23.396993 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 180133us
I20240426 13:21:23.395646 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.411957 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.229995 30601 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:23.411741 25096 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
W20240426 13:21:23.403191 30602 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.567s	user 0.008s	sys 0.000s
I20240426 13:21:23.426651 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:23.235539 30603 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:23.238212 30742 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:23.440452 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.392388 30739 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.384531 30672 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:23.212754 30496 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.401664 30614 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.317552 30704 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:21:23.459772 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.415514 30668 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:23.404364 30409 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:23.435627 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.860s (SENT) [suppressed 3 similar messages]
I20240426 13:21:23.487362 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.504107 25096 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:23.229841 30710 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.235931 30616 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.494290 25105 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:23.514439 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:23.444012 30714 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:23.496701 30614 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:23.595541 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:23.445647 30644 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:23.509577 30704 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:23.512121 30593 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:23.490734 30689 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.552687 30631 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:23.480629 30672 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.499567 30705 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.542985 30742 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:23.618185 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.604933 30668 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:23.650843 25096 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20240426 13:21:23.445941 30602 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.502220 30666 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.651311 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.503145 30201 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.655337 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.639026 30644 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:23.660365 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:23.640967 30604 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:23.665484 25027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:21:23.667871 25027 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:23.669085 25027 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:23.655221 30698 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.226s	user 0.035s	sys 0.029s
I20240426 13:21:23.673034 25023 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20240426 13:21:23.667271 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.675797 25617 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
W20240426 13:21:23.677127 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 236754us
I20240426 13:21:23.678498 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:23.678879 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.682693 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:23.670364 30698 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.241s	user 0.036s	sys 0.029s
I20240426 13:21:23.683709 30707 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:23.686399 30449 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:23.691311 30449 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:23.691661 30449 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:23.691718 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:23.694360 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:23.694588 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2844) took 1189 ms. Trace:
I20240426 13:21:23.695511 25615 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:23.696254 30602 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.696959 30602 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:23.694985 24950 rpcz_store.cc:276] 0426 13:21:22.504634 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.505278 (+   644us) service_pool.cc:225] Handling call
0426 13:21:22.510932 (+  5654us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.692236 (+181304us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:22.692269 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:22.692309 (+    40us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:22.692340 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:23.694365 (+1002025us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54000bf254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:23.694410 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:23.694509 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416485632}
I20240426 13:21:23.702401 24950 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
W20240426 13:21:23.703465 30752 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:21:23.704259 30654 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:23.704897 30750 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20240426 13:21:23.707095 30754 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20240426 13:21:23.709959 30751 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:21:23.714001 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:23.714680 25031 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:23.716630 30756 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:23.721431 30704 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.676617 30670 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.728193 25031 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:23.730109 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 294498us
I20240426 13:21:23.656164 30668 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.742933 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:23.723430 30522 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.688498 30631 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20240426 13:21:23.726893 30753 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20240426 13:21:23.760963 25027 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:23.727311 30404 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.000s
I20240426 13:21:23.689392 30440 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.654713 30742 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.786542 25031 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:23.671797 30644 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.788239 25027 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.740360 30714 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:23.742884 30448 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:23.742898 30707 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:23.800259 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:23.808145 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 156918us
I20240426 13:21:23.706475 30449 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.812172 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.816792 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.818745 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:23.777280 30631 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:23.791328 30651 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
W20240426 13:21:23.840015 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 184732us
I20240426 13:21:23.844684 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:23.849557 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:23.877547 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 84247.4us
I20240426 13:21:23.802796 30710 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.886884 24950 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:23.886723 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:23.820664 30765 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:23.820381 30587 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:23.787189 30660 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.055s
I20240426 13:21:23.832243 30604 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:23.836468 30631 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:23.793537 30404 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.839071 30496 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:23.889227 30671 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.675s	user 0.004s	sys 0.006s
I20240426 13:21:23.812474 30448 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:23.941915 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:23.891729 30616 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.893162 30775 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:23.925501 30689 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.926682 30604 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.965340 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:23.974264 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:23.974891 24950 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:23.931974 30593 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:23.988821 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2833) took 1689 ms. Trace:
I20240426 13:21:23.989173 24953 rpcz_store.cc:276] 0426 13:21:22.299128 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.300439 (+  1311us) service_pool.cc:225] Handling call
0426 13:21:22.306187 (+  5748us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.385320 (+ 79133us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:22.385350 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:22.385387 (+    37us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:22.385416 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:23.988620 (+1603204us) spinlock_profiling.cc:243] Waited 79 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:23.988667 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:23.988774 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":181706880}
I20240426 13:21:23.937927 30705 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:23.996912 25105 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
W20240426 13:21:23.998359 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 111928us
I20240426 13:21:24.028612 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:24.043031 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.043668 24953 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:23.985733 30660 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:23.984935 30666 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:23.963963 30672 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:23.984411 30604 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:23.986797 30482 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:24.041325 30755 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.361s	user 0.000s	sys 0.001s
I20240426 13:21:24.061234 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:24.069867 30683 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:24.073021 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2948) took 1787 ms. Trace:
I20240426 13:21:24.036432 30689 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.076365 30688 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:24.079128 30720 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.085649 30718 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.087461 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2322) took 1015 ms. Trace:
I20240426 13:21:23.990852 30777 raft_consensus.cc:465] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Not starting pre-election -- already a leader
W20240426 13:21:24.058020 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 1118) took 1644 ms (client timeout 2052 ms). Trace:
I20240426 13:21:24.091473 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:24.096253 30522 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.099819 25030 rpcz_store.cc:276] 0426 13:21:22.286304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.286501 (+   197us) service_pool.cc:225] Handling call
0426 13:21:22.320593 (+ 34092us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.995880 (+675287us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.072740 (+1076860us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400241654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.072950 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246065024}
I20240426 13:21:24.046008 30614 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:24.163488 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62565.3us
I20240426 13:21:24.097657 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:24.168609 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:23.953199 30707 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.019335 30593 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.190078 24955 rpcz_store.cc:276] 0426 13:21:23.072429 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.072552 (+   123us) service_pool.cc:225] Handling call
0426 13:21:23.123563 (+ 51011us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.197312 (+ 73749us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.087289 (+889977us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b54000a69d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.087408 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116415872}
I20240426 13:21:24.162923 30767 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.472505 30767 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.474906 30767 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.475517 30767 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.101681 30473 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.476365 30473 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.478749 30473 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.049536 30660 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:24.480772 30660 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.481442 30593 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.482358 30660 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:24.482547 30780 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
W20240426 13:21:24.192523 25624 rpcz_store.cc:269] 0426 13:21:22.413623 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.413737 (+   114us) service_pool.cc:225] Handling call
0426 13:21:24.057790 (+1644053us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.057922 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655705856}
W20240426 13:21:24.293056 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 250117us
W20240426 13:21:24.484396 30790 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
W20240426 13:21:24.391815 30789 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20240426 13:21:24.186034 30651 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:24.185021 30689 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.269183 30786 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:24.264978 24950 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:24.400355 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 56) took 1376 ms. Trace:
I20240426 13:21:24.503551 25099 rpcz_store.cc:276] 0426 13:21:23.024127 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.031446 (+  7319us) service_pool.cc:225] Handling call
0426 13:21:24.400167 (+1368721us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.400280 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":246369024}
I20240426 13:21:24.448486 30614 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:24.303903 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2304) took 1945 ms (client timeout 1860 ms). Trace:
W20240426 13:21:24.507830 24952 rpcz_store.cc:269] 0426 13:21:22.358314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.359660 (+  1346us) service_pool.cc:225] Handling call
0426 13:21:24.303708 (+1944048us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54000bfc54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.303827 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":337471872}
I20240426 13:21:24.298342 30675 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.266721 30705 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.271482 30742 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.518075 30651 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.511819 30799 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20240426 13:21:24.520712 24953 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:24.170176 30668 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.197525 30449 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.521780 30449 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.521545 30675 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.522255 30449 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.168398 30717 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:24.524271 30717 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:24.524662 30717 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:24.525100 30689 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.525604 30689 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.526021 30689 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:24.270025 30755 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:21:24.528210 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:24.530894 25099 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:24.099807 30787 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20240426 13:21:24.360179 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 474529us
I20240426 13:21:24.247586 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2847) took 1738 ms. Trace:
I20240426 13:21:24.249404 25025 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:24.247892 30496 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:24.192464 30720 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:23.964790 30714 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.545356 30742 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:24.534639 24947 rpcz_store.cc:276] 0426 13:21:22.508995 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.510889 (+  1894us) service_pool.cc:225] Handling call
0426 13:21:22.511106 (+   217us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:22.511276 (+   170us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:22.539590 (+ 28314us) log.cc:842] Serialized 14 byte log entry
0426 13:21:23.988824 (+1449234us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b18004a2d80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:21:24.197654 (+208830us) log.cc:842] Serialized 14 byte log entry
0426 13:21:24.206797 (+  9143us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.206864 (+    67us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:24.247331 (+ 40467us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.247416 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.247536 (+   120us) inbound_call.cc:173] Queueing success response
0426 13:21:24.300075 (+ 52539us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b18004a2d80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":239729,"prepare.queue_time_us":28808,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":63609,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":102074880,"thread_start_us":205409,"threads_started":1,"wal-append.queue_time_us":1819353}
W20240426 13:21:24.547266 30797 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:21:24.547399 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.167059 30739 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:24.255650 30644 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.305352 30794 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20240426 13:21:24.548137 30739 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.557536 30794 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:24.557902 30794 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:24.558930 30632 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.131s	user 0.005s	sys 0.005s
I20240426 13:21:24.558601 30794 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.250859 30645 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:24.558971 30739 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.559903 30645 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.561985 30645 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.317813 30667 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:24.562644 30667 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.564605 30667 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.305783 30404 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.450627 30690 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.565529 30690 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.567673 30690 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:24.451599 30788 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
W20240426 13:21:24.452437 30785 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20240426 13:21:24.453245 30737 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:24.448796 25031 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:24.571662 30806 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20240426 13:21:24.301920 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 747) took 1180 ms. Trace:
W20240426 13:21:24.572515 30805 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
I20240426 13:21:24.572901 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 746) took 1474 ms. Trace:
I20240426 13:21:24.573366 25616 rpcz_store.cc:276] 0426 13:21:23.098843 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.099046 (+   203us) service_pool.cc:225] Handling call
0426 13:21:23.191147 (+ 92101us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.191508 (+   361us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.191538 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.191998 (+   460us) log.cc:842] Serialized 26 byte log entry
0426 13:21:23.495223 (+303225us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:23.495280 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:23.495367 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:23.729437 (+234070us) raft_consensus.cc:1671] finished
0426 13:21:23.729457 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.572583 (+843126us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b540071eb50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.572826 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1174,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":167188736,"thread_start_us":302818,"threads_started":1,"wal-append.queue_time_us":469596}
I20240426 13:21:24.456719 30629 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:24.572175 25620 rpcz_store.cc:276] 0426 13:21:23.121092 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.121236 (+   144us) service_pool.cc:225] Handling call
0426 13:21:23.162768 (+ 41532us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:23.600326 (+437558us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.600370 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:23.600425 (+    55us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:23.600469 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.301648 (+701179us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.301702 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.301838 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1005713664}
I20240426 13:21:24.580696 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2156) took 1631 ms. Trace:
I20240426 13:21:24.581213 25102 rpcz_store.cc:276] 0426 13:21:22.949108 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.978305 (+ 29197us) service_pool.cc:225] Handling call
0426 13:21:22.993406 (+ 15101us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:23.056002 (+ 62596us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.580385 (+1524383us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.580631 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142451968}
I20240426 13:21:24.529618 30675 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.528571 30449 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.582731 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:24.457422 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2167) took 1423 ms (client timeout 1799 ms). Trace:
I20240426 13:21:24.583536 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2856) took 1771 ms. Trace:
W20240426 13:21:24.459414 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2953) took 1961 ms (client timeout 2201 ms). Trace:
W20240426 13:21:24.459553 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 964) took 1626 ms (client timeout 1692 ms). Trace:
I20240426 13:21:24.585762 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 970) took 1478 ms. Trace:
W20240426 13:21:24.586365 25024 rpcz_store.cc:269] 0426 13:21:22.498424 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.498903 (+   479us) service_pool.cc:225] Handling call
0426 13:21:24.459254 (+1960351us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54003ae8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.459363 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":491629440}
I20240426 13:21:24.586407 25096 rpcz_store.cc:276] 0426 13:21:23.107121 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.693751 (+586630us) service_pool.cc:225] Handling call
0426 13:21:23.694043 (+   292us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.980350 (+286307us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.980393 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.981012 (+   619us) log.cc:842] Serialized 26 byte log entry
0426 13:21:24.449094 (+468082us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:24.449147 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.585323 (+136176us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.585388 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.585452 (+    64us) raft_consensus.cc:1671] finished
0426 13:21:24.585470 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.585697 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":72484,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7613,"spinlock_wait_cycles":297737472,"thread_start_us":543015,"threads_started":2,"wal-append.queue_time_us":380251}
I20240426 13:21:24.587777 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:24.588588 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.589146 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2551) took 1090 ms. Trace:
I20240426 13:21:24.469219 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.589891 30807 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.501343 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:24.591281 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.300963 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2966) took 1509 ms. Trace:
I20240426 13:21:24.592209 25023 rpcz_store.cc:276] 0426 13:21:22.791288 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.825958 (+ 34670us) service_pool.cc:225] Handling call
0426 13:21:22.826275 (+   317us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.676243 (+849968us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.300674 (+624431us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.300898 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1946595584}
W20240426 13:21:24.592723 25100 rpcz_store.cc:269] 0426 13:21:22.833033 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.833754 (+   721us) service_pool.cc:225] Handling call
0426 13:21:24.459390 (+1625636us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.459485 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":835477248}
I20240426 13:21:24.594395 25023 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:24.594815 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 972) took 1486 ms. Trace:
I20240426 13:21:24.595757 30666 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
W20240426 13:21:24.591753 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 90523.4us
I20240426 13:21:24.596346 30666 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:24.597476 30666 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.600158 30720 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.595371 25100 rpcz_store.cc:276] 0426 13:21:23.108711 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.593917 (+1485206us) service_pool.cc:225] Handling call
0426 13:21:24.594227 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.594417 (+   190us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.594453 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.594521 (+    68us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:24.594575 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.594640 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.594755 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.589807 25105 rpcz_store.cc:276] 0426 13:21:23.499145 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.511336 (+ 12191us) service_pool.cc:225] Handling call
0426 13:21:24.588982 (+1077646us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.589092 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":985500416}
W20240426 13:21:24.501192 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.006s	sys 0.010s
I20240426 13:21:24.590198 25024 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:21:24.420600 30793 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.509017 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.509389 25617 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:24.512195 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.559420 30632 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:24.513995 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 485427us
I20240426 13:21:24.514588 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:24.516945 30705 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.520864 30799 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:24.094681 30482 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:24.587778 30409 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.988s	user 0.004s	sys 0.004s
I20240426 13:21:24.538828 25030 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:24.509690 30786 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.545107 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:24.629323 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 380538us
I20240426 13:21:24.547995 24947 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:24.485672 30767 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.633318 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.554668 25099 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:24.635596 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:24.629050 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122352us
W20240426 13:21:24.555953 25028 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.010s	sys 0.012s
W20240426 13:21:24.583740 25101 rpcz_store.cc:269] 0426 13:21:23.033658 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.048289 (+ 14631us) service_pool.cc:225] Handling call
0426 13:21:24.457268 (+1408979us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b540009d3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.457371 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56814336}
W20240426 13:21:24.636744 25028 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.011s	sys 0.012s
I20240426 13:21:24.589172 25096 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:24.590525 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2157) took 1639 ms. Trace:
I20240426 13:21:24.637326 25103 rpcz_store.cc:276] 0426 13:21:22.951308 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.952143 (+   835us) service_pool.cc:225] Handling call
0426 13:21:23.001712 (+ 49569us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:23.002425 (+   713us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.590347 (+1587922us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b54003ef8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.590474 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20607360}
I20240426 13:21:24.638538 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.496651 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2973) took 1427 ms. Trace:
I20240426 13:21:24.639251 25022 rpcz_store.cc:276] 0426 13:21:23.069077 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.174028 (+104951us) service_pool.cc:225] Handling call
0426 13:21:23.174323 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.543517 (+369194us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.543552 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.544069 (+   517us) log.cc:842] Serialized 26 byte log entry
0426 13:21:23.839333 (+295264us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:23.839389 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.478810 (+639421us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54004155d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:24.478876 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.485267 (+  6391us) raft_consensus.cc:1671] finished
0426 13:21:24.485286 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.496585 (+ 11299us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":26552,"spinlock_wait_cycles":811974016,"thread_start_us":203285,"threads_started":1,"wal-append.queue_time_us":397664}
W20240426 13:21:24.493597 24949 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.008s	sys 0.011s
W20240426 13:21:24.641003 24949 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.008s	sys 0.011s
I20240426 13:21:24.360396 30591 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:24.504629 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.005s	sys 0.011s
W20240426 13:21:24.649055 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.006s	sys 0.011s
W20240426 13:21:24.649375 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 757366us
I20240426 13:21:24.641777 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2328) took 1556 ms. Trace:
I20240426 13:21:24.607244 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2545) took 1445 ms. Trace:
I20240426 13:21:24.665844 25105 rpcz_store.cc:276] 0426 13:21:23.162297 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.606553 (+1444256us) service_pool.cc:225] Handling call
0426 13:21:24.606789 (+   236us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.606945 (+   156us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.606968 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.607004 (+    36us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:24.607050 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.607095 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.607177 (+    82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.529951 30614 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:24.613485 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:24.588611 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2159) took 1635 ms. Trace:
I20240426 13:21:24.709214 25098 rpcz_store.cc:276] 0426 13:21:22.952826 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.952957 (+   131us) service_pool.cc:225] Handling call
0426 13:21:24.465617 (+1512660us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.465820 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.465853 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.465901 (+    48us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:24.465941 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.466008 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.588452 (+122444us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540012cad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.588561 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3477885312}
I20240426 13:21:24.710793 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:24.583971 24954 rpcz_store.cc:276] 0426 13:21:22.812184 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.992264 (+180080us) service_pool.cc:225] Handling call
0426 13:21:22.992618 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.069541 (+ 76923us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.069569 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.069937 (+   368us) log.cc:842] Serialized 26 byte log entry
0426 13:21:23.336349 (+266412us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:23.399110 (+ 62761us) log.cc:842] Serialized 14 byte log entry
0426 13:21:23.400354 (+  1244us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:23.400448 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.583351 (+1182903us) raft_consensus.cc:1671] finished
0426 13:21:24.583371 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.583484 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123983,"prepare.queue_time_us":60199,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":3370,"raft.queue_time_us":381880,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":18584,"thread_start_us":210396,"threads_started":1,"wal-append.queue_time_us":536481}
W20240426 13:21:24.507481 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.002s	sys 0.002s
W20240426 13:21:24.712343 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.050s	user 0.003s	sys 0.003s
I20240426 13:21:24.617080 24950 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:24.547958 30689 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.529532 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.628129 24955 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20240426 13:21:24.637619 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:24.607723 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.007s	sys 0.010s
I20240426 13:21:24.718730 24953 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:24.616091 30644 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.625311 30601 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.646673 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2188) took 1601 ms. Trace:
I20240426 13:21:24.646905 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 755) took 1239 ms. Trace:
W20240426 13:21:24.692323 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 138841us
I20240426 13:21:24.625517 30799 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:24.718772 25099 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:24.755620 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:24.722939 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.646426 24948 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:24.756397 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.650755 24949 rpcz_store.cc:276] 0426 13:21:23.085040 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.400718 (+315678us) service_pool.cc:225] Handling call
0426 13:21:23.425747 (+ 25029us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:23.426019 (+   272us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:23.822010 (+395991us) log.cc:842] Serialized 14 byte log entry
0426 13:21:24.641343 (+819333us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.641404 (+    61us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:24.641462 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.641534 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.641700 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7443,"prepare.queue_time_us":55137,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":192839,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":1155509,"threads_started":3,"wal-append.queue_time_us":124859}
I20240426 13:21:24.731595 25616 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
W20240426 13:21:24.635538 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 116078us
I20240426 13:21:24.734122 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:24.736549 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.760633 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.761716 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:24.765617 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:24.766273 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:24.753607 30668 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.753917 30681 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.767211 30668 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.767441 30681 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.767990 25031 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:24.769151 30668 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.769629 30720 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.769927 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.770133 30720 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.770920 30496 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.771085 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.771420 30496 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.739517 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:24.739588 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.740514 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.740523 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.601045 30807 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.692932 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:24.744539 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:24.745357 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2162) took 1785 ms (client timeout 2234 ms). Trace:
I20240426 13:21:24.745486 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.634109 30409 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.741969 25028 rpcz_store.cc:276] 0426 13:21:23.045040 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.049460 (+  4420us) service_pool.cc:225] Handling call
0426 13:21:23.049756 (+   296us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.068133 (+ 18377us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.068163 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.124286 (+ 56123us) log.cc:842] Serialized 26 byte log entry
0426 13:21:23.410537 (+286251us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:23.849129 (+438592us) log.cc:842] Serialized 14 byte log entry
0426 13:21:24.566090 (+716961us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b48000b19d8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:21:24.637107 (+ 71017us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.637179 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.637216 (+    37us) raft_consensus.cc:1671] finished
0426 13:21:24.646467 (+  9251us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.646606 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64936,"prepare.queue_time_us":65533,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":5009,"raft.queue_time_us":199937,"raft.run_cpu_time_us":1878,"raft.run_wall_time_us":524512,"spinlock_wait_cycles":14344448,"thread_start_us":1388800,"threads_started":3,"wal-append.queue_time_us":438162}
I20240426 13:21:24.739035 30602 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:24.773396 24951 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:24.773726 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.754122 29429 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.754315 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.766813 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.738687 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.742039 25027 rpcz_store.cc:276] 0426 13:21:23.407150 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.839913 (+432763us) service_pool.cc:225] Handling call
0426 13:21:23.840349 (+   436us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.881727 (+ 41378us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:23.881941 (+   214us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.885056 (+  3115us) log.cc:842] Serialized 26 byte log entry
0426 13:21:24.595219 (+710163us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b18000aed00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291
0426 13:21:24.595353 (+   134us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:24.595657 (+   304us) log.cc:842] Serialized 14 byte log entry
0426 13:21:24.634209 (+ 38552us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.634275 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.634312 (+    37us) raft_consensus.cc:1671] finished
0426 13:21:24.646740 (+ 12428us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.646856 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135559,"prepare.queue_time_us":6030,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":236080,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":101661,"spinlock_wait_cycles":51119488,"thread_start_us":341120,"threads_started":2,"wal-append.queue_time_us":492152}
I20240426 13:21:24.739226 30678 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.739784 30730 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.784993 30817 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:24.834518 30730 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.834970 30817 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.762216 30601 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.785084 30716 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:24.837004 25616 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:24.837213 30730 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.837515 30781 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.837980 30601 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.838057 30781 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.626094 30793 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.772539 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.734678 30690 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.760464 30616 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.797612 30675 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:24.769151 30681 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.756284 30644 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.762470 30591 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:24.766837 30448 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.779598 24952 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:24.779645 25103 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:21:24.752657 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:24.811015 30691 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:24.780371 25104 rpcz_store.cc:269] 0426 13:21:22.959905 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.001044 (+ 41139us) service_pool.cc:225] Handling call
0426 13:21:24.745201 (+1744157us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:24.745310 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69537024}
I20240426 13:21:24.772007 30720 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.740597 30614 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.624191 30798 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.161s
I20240426 13:21:24.814261 30705 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.879635 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:24.814572 30717 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.782671 30778 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.187s
I20240426 13:21:24.815810 30496 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.784457 30409 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.816781 25099 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:24.818624 30786 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.743134 30680 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:24.783146 30801 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20240426 13:21:24.785976 24954 raft_consensus.cc:2396] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20240426 13:21:24.796734 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.773701 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:24.799691 25615 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:24.799000 29429 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.853565 30678 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.848273 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:24.902096 25022 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:24.848659 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:24.849105 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.851053 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2547) took 1588 ms. Trace:
I20240426 13:21:24.882846 30767 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.886044 30817 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:24.857057 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90845.6us
I20240426 13:21:24.743420 30667 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
W20240426 13:21:24.784700 30782 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20240426 13:21:24.844918 24955 raft_consensus.cc:2362] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15. Re-sending same reply.
I20240426 13:21:24.947866 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:24.859016 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:24.947240 30767 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.862830 25101 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:21:24.812801 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:24.862922 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103172us
I20240426 13:21:24.953106 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:24.864599 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2550) took 1419 ms. Trace:
I20240426 13:21:24.954627 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2549) took 1544 ms. Trace:
I20240426 13:21:24.954715 25103 rpcz_store.cc:276] 0426 13:21:23.444765 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.863643 (+1418878us) service_pool.cc:225] Handling call
0426 13:21:24.864007 (+   364us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.864257 (+   250us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.864293 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.864346 (+    53us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:24.864397 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.864449 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.864547 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.866021 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2553) took 1315 ms. Trace:
I20240426 13:21:24.956400 24949 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:24.750458 24947 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:24.954044 30801 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20240426 13:21:24.840180 30602 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:24.878304 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 125745us
I20240426 13:21:24.968149 30602 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:24.968429 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:24.969380 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:24.775719 30473 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:24.970474 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:24.970674 30473 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.971550 30739 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:24.971983 30739 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.972378 30739 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:24.972721 30473 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:24.860574 30625 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.758s	user 0.010s	sys 0.002s
I20240426 13:21:24.973412 30625 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.975268 30625 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.866358 30671 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:24.876405 30819 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:24.976056 30671 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:24.976246 30819 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:24.977409 30819 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.977669 30671 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.980738 30625 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:24.976672 25097 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:24.982422 30409 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:24.983088 30409 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:24.983405 30409 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:24.983731 30409 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:24.948057 30632 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:24.986842 30644 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:24.837284 30716 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:24.795202 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61161.8us
I20240426 13:21:24.903622 25098 rpcz_store.cc:276] 0426 13:21:23.262306 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.780932 (+1518626us) service_pool.cc:225] Handling call
0426 13:21:24.850401 (+ 69469us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.850613 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.850654 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.850750 (+    96us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:24.850794 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.850864 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.850990 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.994537 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:24.994717 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:24.848019 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2189) took 1662 ms. Trace:
I20240426 13:21:24.995885 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:24.903170 24950 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:24.860497 30799 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.906224 29429 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:24.932426 30253 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:24.995355 25029 rpcz_store.cc:276] 0426 13:21:23.185414 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.653149 (+467735us) service_pool.cc:225] Handling call
0426 13:21:23.653399 (+   250us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:23.745728 (+ 92329us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:21:23.745864 (+   136us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:23.748807 (+  2943us) log.cc:842] Serialized 26 byte log entry
0426 13:21:24.718950 (+970143us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:24.719022 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.719134 (+   112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:24.847843 (+128709us) raft_consensus.cc:1671] finished
0426 13:21:24.847864 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.847971 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44405,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":57,"thread_start_us":1002455,"threads_started":2,"wal-append.queue_time_us":968162}
W20240426 13:21:24.934834 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 161211us
I20240426 13:21:25.002610 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:24.860381 30690 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:24.847507 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:24.886533 30482 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:25.011121 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 163680us
I20240426 13:21:25.012682 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.014075 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:24.946619 24954 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:24.860124 30616 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:24.737026 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.011s	sys 0.013s
I20240426 13:21:24.976327 30816 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
W20240426 13:21:25.017988 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.167s	user 0.012s	sys 0.013s
I20240426 13:21:24.956437 25104 rpcz_store.cc:276] 0426 13:21:23.550878 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.865288 (+1314410us) service_pool.cc:225] Handling call
0426 13:21:24.865566 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.865728 (+   162us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.865761 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.865805 (+    44us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:24.865839 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.865887 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.865976 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.958380 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1026) took 1067 ms. Trace:
I20240426 13:21:25.018422 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:25.018689 24949 rpcz_store.cc:276] 0426 13:21:23.890734 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.935945 (+1045211us) service_pool.cc:225] Handling call
0426 13:21:24.956246 (+ 20301us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.958176 (+  1930us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.958334 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.999364 30253 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:24.960258 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:24.957093 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2873) took 1031 ms. Trace:
I20240426 13:21:24.951558 30767 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:24.967687 30817 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:24.950148 30629 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:24.989023 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2554) took 1368 ms. Trace:
I20240426 13:21:24.998072 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2874) took 1063 ms. Trace:
I20240426 13:21:25.043573 25103 rpcz_store.cc:276] 0426 13:21:23.620670 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.956008 (+1335338us) service_pool.cc:225] Handling call
0426 13:21:24.967931 (+ 11923us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.988698 (+ 20767us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.988726 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.988767 (+    41us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:24.988798 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.988851 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.988960 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.044473 24950 rpcz_store.cc:276] 0426 13:21:23.934713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.997210 (+1062497us) service_pool.cc:225] Handling call
0426 13:21:24.997535 (+   325us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.997741 (+   206us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.997766 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.997815 (+    49us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:24.997856 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.997917 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:24.998029 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:24.896907 30680 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.045831 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 975) took 1235 ms. Trace:
I20240426 13:21:25.007826 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:24.947770 30815 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:21:25.027760 30660 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.545s	user 0.001s	sys 0.007s
I20240426 13:21:24.957863 24952 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:24.955112 25101 rpcz_store.cc:276] 0426 13:21:23.410357 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.799165 (+1388808us) service_pool.cc:225] Handling call
0426 13:21:24.860178 (+ 61013us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.954408 (+ 94230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.954574 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.022359 24947 rpcz_store.cc:276] 0426 13:21:23.925247 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.547160 (+621913us) service_pool.cc:225] Handling call
0426 13:21:24.957039 (+409879us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:24.950662 30683 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.822s	user 0.008s	sys 0.002s
I20240426 13:21:24.988847 30614 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:25.018435 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.16714e+06us
I20240426 13:21:25.024765 24949 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20240426 13:21:25.025413 30681 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:25.037209 25624 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:25.036545 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 76365.4us
I20240426 13:21:25.069015 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.042888 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:25.006299 30778 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20240426 13:21:25.046247 25103 rpcz_store.cc:276] 0426 13:21:23.810762 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.044935 (+1234173us) service_pool.cc:225] Handling call
0426 13:21:25.045304 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.045501 (+   197us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.045524 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.045587 (+    63us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:25.045634 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.045687 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.045783 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.017167 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.006057 30496 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:25.050874 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1027) took 1159 ms. Trace:
I20240426 13:21:25.073640 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 977) took 1261 ms. Trace:
I20240426 13:21:25.080001 24954 rpcz_store.cc:276] 0426 13:21:23.890991 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.938750 (+1047759us) service_pool.cc:225] Handling call
0426 13:21:24.939107 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.050651 (+111544us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.050825 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:21:25.051679 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43939.9us
I20240426 13:21:25.020148 30632 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:25.019836 30813 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:25.019677 30665 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:25.019958 30742 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:25.064527 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2875) took 1120 ms. Trace:
I20240426 13:21:25.091984 24949 rpcz_store.cc:276] 0426 13:21:23.944136 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.019478 (+1075342us) service_pool.cc:225] Handling call
0426 13:21:25.024564 (+  5086us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:25.064202 (+ 39638us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.064458 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.069679 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.070281 25028 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:25.094211 25028 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.070972 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.009301 30720 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
I20240426 13:21:25.076267 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.053669 30817 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.114814 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.115540 24950 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:21:25.115955 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.116446 30817 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.076706 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 976) took 1265 ms. Trace:
I20240426 13:21:25.117374 25101 rpcz_store.cc:276] 0426 13:21:23.811068 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.057352 (+1246284us) service_pool.cc:225] Handling call
0426 13:21:25.076247 (+ 18895us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.076409 (+   162us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.076433 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.076471 (+    38us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:25.076501 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.076561 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.076650 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.119303 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2175) took 1303 ms. Trace:
W20240426 13:21:25.077046 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59950.3us
I20240426 13:21:25.120836 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.077720 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:25.053807 30448 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:25.124166 30690 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.124547 30448 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:25.125483 30448 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.126562 30767 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.064364 30717 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.127290 30642 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:25.054085 30799 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:25.128414 30642 raft_consensus.cc:2741] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20240426 13:21:25.129684 30670 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.129772 30642 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:25.080406 25103 rpcz_store.cc:276] 0426 13:21:23.811836 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.072712 (+1260876us) service_pool.cc:225] Handling call
0426 13:21:25.073054 (+   342us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.073255 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.073291 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.073344 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:25.073388 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.073453 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.073577 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.131826 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 979) took 1284 ms. Trace:
I20240426 13:21:25.132259 25103 rpcz_store.cc:276] 0426 13:21:23.847398 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.131059 (+1283661us) service_pool.cc:225] Handling call
0426 13:21:25.131334 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.131491 (+   157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.131515 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.131577 (+    62us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:25.131625 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.131677 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.131782 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.133390 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.050163 30816 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:25.000113 30809 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20240426 13:21:25.027828 30739 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.084380 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:25.143077 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61250.7us
I20240426 13:21:25.057986 30714 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:21:25.063076 30631 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.123s	user 0.006s	sys 0.006s
I20240426 13:21:25.082072 30666 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:25.095809 30742 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:25.095991 30672 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
W20240426 13:21:25.096146 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53337.8us
I20240426 13:21:25.078068 30593 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.097594 30616 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.103142 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2870) took 1213 ms. Trace:
I20240426 13:21:25.104625 25028 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.152370 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.153158 25028 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:25.106245 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.059945 30815 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.053580 30629 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.111043 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.113133 24956 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:25.133993 25103 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:25.163658 25027 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:25.164206 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.119812 30807 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.137790 25624 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20240426 13:21:25.138885 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1022) took 2035 ms. Trace:
I20240426 13:21:25.130383 30642 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:25.140206 30817 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:25.140895 30691 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.141093 30315 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:25.152192 30742 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:25.149056 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:25.186760 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:25.150092 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.189067 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.190434 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.192250 30614 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:25.192824 30614 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:25.193274 30614 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:25.152313 24953 rpcz_store.cc:276] 0426 13:21:23.889841 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.761584 (+871743us) service_pool.cc:225] Handling call
0426 13:21:24.795890 (+ 34306us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:24.798329 (+  2439us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.798359 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:24.798738 (+   379us) log.cc:842] Serialized 26 byte log entry
0426 13:21:24.992904 (+194166us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:24.993356 (+   452us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.993452 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:25.102884 (+109432us) raft_consensus.cc:1671] finished
0426 13:21:25.102915 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.103071 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1909,"raft.queue_time_us":15060,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"thread_start_us":182051,"threads_started":1,"wal-append.queue_time_us":290485}
I20240426 13:21:25.119643 25101 rpcz_store.cc:276] 0426 13:21:23.815877 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.118465 (+1302588us) service_pool.cc:225] Handling call
0426 13:21:25.118759 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.118939 (+   180us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.118976 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.119049 (+    73us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:25.119093 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.119148 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.119250 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.167762 25103 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:25.171797 24951 rpcz_store.cc:276] 0426 13:21:23.102976 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.714316 (+611340us) service_pool.cc:225] Handling call
0426 13:21:23.747633 (+ 33317us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.798350 (+1050717us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.138563 (+340213us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.138779 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2358353280}
I20240426 13:21:25.146160 30714 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:25.175705 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.212858 25022 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:25.147361 30496 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:25.034162 30409 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.130285 30670 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.184974 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.087594 30602 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.146024 30601 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20240426 13:21:25.167822 30730 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.214119 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.147156 30720 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20240426 13:21:25.168758 30672 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.227077 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.191804 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.213208 30707 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.230566 25027 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:21:25.231812 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.216054 30811 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:25.245496 25618 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:25.259929 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 116343us
I20240426 13:21:25.149016 30631 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.247716 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.246670 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:25.169683 30593 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.256290 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.256433 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.216493 30714 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:25.278136 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.270459 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:25.270879 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2863) took 2176 ms (client timeout 2850 ms). Trace:
I20240426 13:21:25.278656 25099 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
W20240426 13:21:25.279145 24948 rpcz_store.cc:269] 0426 13:21:23.094842 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.220064 (+125222us) service_pool.cc:225] Handling call
0426 13:21:25.267577 (+2047513us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54000a64d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:25.267751 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3248082176}
I20240426 13:21:25.279645 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.280762 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:25.272744 25617 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:25.274145 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.272594 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.240793 30690 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:25.261831 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 172134us
I20240426 13:21:25.272562 25027 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:25.287369 25027 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:25.287765 30654 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:25.282265 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.288370 30654 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.289345 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.291532 30556 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:25.292143 30556 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:25.292676 30556 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:25.293162 30556 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.293211 30654 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.296262 30522 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:25.296783 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:25.297169 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2982) took 1785 ms. Trace:
I20240426 13:21:25.297652 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:25.298339 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2544) took 2139 ms. Trace:
I20240426 13:21:25.298682 25100 rpcz_store.cc:276] 0426 13:21:23.158995 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.603723 (+1444728us) service_pool.cc:225] Handling call
0426 13:21:24.637362 (+ 33639us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.637548 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.637578 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.637619 (+    41us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:24.637653 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:24.637702 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.298193 (+660491us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b54003e12d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:25.298289 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76714368}
I20240426 13:21:25.302089 30692 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:25.304306 30201 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:25.184968 30691 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.184770 30817 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.307466 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.308110 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.308387 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.297499 25030 rpcz_store.cc:276] 0426 13:21:23.511769 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.261548 (+749779us) service_pool.cc:225] Handling call
0426 13:21:24.270975 (+  9427us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.629246 (+358271us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.296965 (+667719us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.297114 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615437184}
I20240426 13:21:25.299901 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.388357 25099 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
W20240426 13:21:25.390286 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 320037us
I20240426 13:21:25.391829 25098 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:25.212286 30315 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.393103 25022 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:25.394922 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:25.395591 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85192.8us
I20240426 13:21:25.412256 25030 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:25.400045 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2164) took 2424 ms. Trace:
I20240426 13:21:25.413007 25097 rpcz_store.cc:276] 0426 13:21:22.967965 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.017515 (+ 49550us) service_pool.cc:225] Handling call
0426 13:21:23.017771 (+   256us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.982209 (+1964438us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.285806 (+303597us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.391866 (+106060us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4504593280}
I20240426 13:21:25.257952 30716 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.278646 30666 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:25.402174 25096 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20240426 13:21:25.307430 30604 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.184s	user 0.008s	sys 0.004s
I20240426 13:21:25.296847 30522 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:25.416185 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.422420 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 980) took 1567 ms. Trace:
I20240426 13:21:25.478086 25100 rpcz_store.cc:276] 0426 13:21:23.855413 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.407607 (+1552194us) service_pool.cc:225] Handling call
0426 13:21:25.421819 (+ 14212us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.422026 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.422063 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.422128 (+    65us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:25.422174 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.422239 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.422358 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.307803 30675 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:25.435392 25025 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:25.275969 30811 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.474121 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2872) took 1578 ms. Trace:
I20240426 13:21:25.475193 25097 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20240426 13:21:25.532742 30811 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.534358 25097 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20240426 13:21:25.535665 25097 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:25.536581 25097 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:25.537398 25097 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.538290 30409 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.538434 25097 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:25.539124 30409 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:21:25.540882 30602 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.542667 30691 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:25.544806 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2321) took 2473 ms (client timeout 1506 ms). Trace:
W20240426 13:21:25.545346 24956 rpcz_store.cc:269] 0426 13:21:23.071854 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.072156 (+   302us) service_pool.cc:225] Handling call
0426 13:21:25.544585 (+2472429us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:25.544705 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4648600320}
I20240426 13:21:25.553620 30629 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.402053 30680 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.555135 30645 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:25.558275 30614 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.242545 30816 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.560998 25097 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:25.401379 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:25.485731 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 209018us
W20240426 13:21:25.486171 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 266413us
I20240426 13:21:25.246639 30730 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.501209 30556 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.416147 30778 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
W20240426 13:21:25.502337 30825 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.256s	user 0.000s	sys 0.000s
I20240426 13:21:25.505949 30817 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.508152 30672 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.510197 30315 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:25.510697 30829 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.265s	user 0.000s	sys 0.000s
I20240426 13:21:25.515369 30704 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.522265 30631 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.525451 30781 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:25.527653 30824 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.285s	user 0.000s	sys 0.000s
I20240426 13:21:25.528661 30794 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:25.532231 24955 rpcz_store.cc:276] 0426 13:21:23.895780 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.949983 (+1054203us) service_pool.cc:225] Handling call
0426 13:21:24.950274 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.950450 (+   176us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:24.950701 (+   251us) log.cc:842] Serialized 14 byte log entry
0426 13:21:25.473402 (+522701us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:25.473472 (+    70us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:25.473537 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.473903 (+   366us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.474052 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3644,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":50414,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":10037,"thread_start_us":522264,"threads_started":2,"wal-append.queue_time_us":285991}
I20240426 13:21:25.565006 25029 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:25.571346 25097 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 16. Ops: []
I20240426 13:21:25.572024 25022 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:25.262257 30707 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.501411 30850 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:25.578117 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.418718 30792 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:25.585199 30253 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.496781 30604 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.588937 30793 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.592927 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.599728 30593 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.603221 30670 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.604462 25615 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:25.673626 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 382594us
I20240426 13:21:25.609508 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:25.605468 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2172) took 1952 ms. Trace:
I20240426 13:21:25.610430 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.610989 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.612541 25621 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:25.391667 30690 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.676988 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:25.676995 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 67546us
I20240426 13:21:25.628589 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.631433 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.581712 30778 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:25.615809 30739 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.701246 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.702533 25623 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:25.644357 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1456) took 1991 ms. Trace:
I20240426 13:21:25.642633 25101 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:25.587357 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.642153 30850 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:25.639034 30409 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:21:25.650238 30799 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.000s
I20240426 13:21:25.651552 30792 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.726016 30718 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:25.726956 30792 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:25.710465 30409 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:25.728072 30409 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.610129 30681 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20240426 13:21:25.729555 30681 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:25.729952 30681 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.730418 30642 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.730764 30681 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.396129 30815 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.611457 30716 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20240426 13:21:25.652597 30253 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:25.681064 25617 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:25.686355 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.733573 24955 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:25.685616 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.703507 25097 rpcz_store.cc:276] 0426 13:21:23.653015 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.643073 (+1990058us) service_pool.cc:225] Handling call
0426 13:21:25.644308 (+  1235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:25.738328 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.739183 24955 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:25.740432 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.690346 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:25.711746 25028 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.000s
I20240426 13:21:25.707347 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:25.714565 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.744894 25028 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:21:25.715798 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:25.717005 30831 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20240426 13:21:25.687717 30614 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.631327 30704 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.675223 25104 rpcz_store.cc:276] 0426 13:21:23.653312 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.018776 (+1365464us) service_pool.cc:225] Handling call
0426 13:21:25.019078 (+   302us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:25.019405 (+   327us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.019431 (+    26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:25.265603 (+246172us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b2c0012e098. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:25.266202 (+   599us) log.cc:842] Serialized 52 byte log entry
0426 13:21:25.293938 (+ 27736us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:25.293972 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.294033 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:25.564406 (+270373us) raft_consensus.cc:1671] finished
0426 13:21:25.564434 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.605409 (+ 40975us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1923,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":21238,"spinlock_wait_cycles":2937088,"thread_start_us":27379,"threads_started":1,"wal-append.queue_time_us":246059}
I20240426 13:21:25.719707 30690 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.721799 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:25.722414 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.758984 24956 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:25.723973 30707 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.701730 30776 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:25.725188 30678 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.731464 30642 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:25.696806 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2882) took 1003 ms. Trace:
I20240426 13:21:25.775844 24950 rpcz_store.cc:276] 0426 13:21:24.692948 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.052154 (+359206us) service_pool.cc:225] Handling call
0426 13:21:25.052457 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.115881 (+ 63424us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.696535 (+580654us) spinlock_profiling.cc:243] Waited 62.8 ms on lock 0x7b54000a69d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.696740 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144378624}
I20240426 13:21:25.656867 30201 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.748395 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.754518 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:25.740126 30742 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.000s
W20240426 13:21:25.769590 30846 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:21:25.770032 30781 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:25.770651 25031 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:25.770890 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2546) took 2566 ms. Trace:
I20240426 13:21:25.804469 25105 rpcz_store.cc:276] 0426 13:21:23.204454 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.760518 (+1556064us) service_pool.cc:225] Handling call
0426 13:21:24.773968 (+ 13450us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:24.877188 (+103220us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:24.877233 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:24.877288 (+    55us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:24.877333 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.770704 (+893371us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.770746 (+    42us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:25.770845 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236865408}
I20240426 13:21:25.771281 30604 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:25.771791 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1136) took 1926 ms. Trace:
I20240426 13:21:25.771628 30778 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.772848 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.757180 30716 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:25.759680 30681 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:25.791458 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.794474 24955 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:25.806486 25623 rpcz_store.cc:276] 0426 13:21:23.845855 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:23.846014 (+   159us) service_pool.cc:225] Handling call
0426 13:21:23.887568 (+ 41554us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.702920 (+1815352us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:25.771567 (+ 68647us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:25.771736 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4173545344}
I20240426 13:21:25.798702 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:25.804175 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.808600 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.813514 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:25.752064 30714 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:25.791725 30776 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:25.816329 25022 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20240426 13:21:25.818068 25028 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:25.776835 30793 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
W20240426 13:21:25.815511 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.037s	sys 0.055s
W20240426 13:21:25.827032 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.037s	sys 0.055s
I20240426 13:21:25.828426 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:25.829113 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 526554us
I20240426 13:21:25.830578 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.833371 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.825273 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.835963 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.809144 30716 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:25.830185 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.028s	sys 0.074s
W20240426 13:21:25.841681 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.028s	sys 0.074s
W20240426 13:21:25.842365 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 553014us
I20240426 13:21:25.844879 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.845970 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.847919 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.849172 25099 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [13.7-16.8]
I20240426 13:21:25.849893 24950 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:25.852362 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.798314 30201 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.854775 30201 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:25.856142 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:25.864593 25099 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20240426 13:21:25.866518 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.837904 30604 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20240426 13:21:25.844179 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:25.819073 30632 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.002s	sys 0.016s
I20240426 13:21:25.876106 25623 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20240426 13:21:25.798161 30742 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:25.805127 30650 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20240426 13:21:25.883535 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.836349 30778 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:25.891101 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.870187 30670 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:25.892940 25022 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:25.872081 30716 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:25.877513 30632 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.002s	sys 0.016s
I20240426 13:21:25.887986 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:25.897104 25024 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:25.897976 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.898932 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:25.899307 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:25.904747 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.837647 30678 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20240426 13:21:25.911218 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.916829 25031 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:25.917412 25615 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:25.918591 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 937671us
I20240426 13:21:25.913168 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.914216 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.916388 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.928521 25022 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:25.914844 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.929546 25105 raft_consensus.cc:2396] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20240426 13:21:25.944681 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:25.944823 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:25.947145 25101 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20240426 13:21:25.950505 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:25.884451 30781 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20240426 13:21:25.915149 30201 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:25.963471 30650 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:25.965377 30404 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:25.969529 30705 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:25.981499 25027 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20240426 13:21:25.983737 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 413458us
I20240426 13:21:25.999531 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:25.900779 30670 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.002259 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:25.954548 30593 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20240426 13:21:25.982527 30201 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:25.948155 30793 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20240426 13:21:26.027755 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:26.027901 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 457292us
I20240426 13:21:25.998402 30705 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:26.032299 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.032809 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:25.950242 30776 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.039515 30496 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20240426 13:21:26.041203 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.042013 30816 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:26.042431 30816 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.042800 30816 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.048216 30759 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:26.048691 30759 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.049069 30759 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:26.020843 30315 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20240426 13:21:26.052852 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.056109 30714 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.056694 30604 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.024221 30631 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20240426 13:21:26.060792 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.061244 30799 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:26.061532 30847 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:26.061640 30799 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:26.062178 30847 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:26.062542 30799 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.062974 25026 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:26.063046 30847 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.065232 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:26.065567 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2962) took 3290 ms (client timeout 1521 ms). Trace:
I20240426 13:21:26.020613 30774 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:26.068501 25026 rpcz_store.cc:269] 0426 13:21:22.775460 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:22.776097 (+   637us) service_pool.cc:225] Handling call
0426 13:21:26.065352 (+3289255us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:26.065485 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7482198912}
I20240426 13:21:26.069151 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.025799 30201 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:26.080992 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.082584 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.084632 24955 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:26.086997 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.088052 25100 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:26.031960 30792 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:26.104565 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.106128 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:26.063953 30799 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:26.109593 25621 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:26.064332 30705 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:26.067921 30631 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.066248 30812 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.117635 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:26.142160 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 71279.9us
I20240426 13:21:26.142575 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.143659 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.143895 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.144292 24956 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:26.145227 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.145532 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.145727 30602 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:26.146164 30602 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:26.146284 24953 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:26.146557 30602 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.076601 30315 consensus_queue.cc:1225] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20240426 13:21:26.151696 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82611.7us
I20240426 13:21:26.152737 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:26.153398 24956 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:26.145550 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.153429 30602 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.156168 30602 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.156221 30730 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.156412 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.157418 30774 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:26.157796 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:26.159035 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.057121 30670 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.077661 30604 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.161708 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.162209 25101 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:26.162820 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.163010 24956 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:26.163965 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.169737 30812 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.243731 30253 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:26.240835 30781 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:26.178436 25104 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20240426 13:21:26.246847 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82941us
I20240426 13:21:26.192662 24953 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:26.248060 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.249411 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.182070 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:26.255652 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.143802 30681 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.268865 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.269584 24952 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:26.297255 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47944.5us
I20240426 13:21:26.298476 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.170362 30768 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.169054 30496 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:26.330768 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.158865 30742 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.325132 30718 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:26.328620 30848 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:26.333108 30642 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:26.336452 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 174827us
I20240426 13:21:26.337215 30759 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.337731 30681 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:26.241747 30614 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:26.340214 30774 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:26.350286 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.363421 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.364039 24953 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:26.350623 24952 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:26.171917 30680 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.244364 30781 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:26.251927 30845 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.244341 30253 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.170013 30792 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.328966 30556 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:26.388540 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:26.390605 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.349288 30602 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.255931 30816 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.422714 24951 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20240426 13:21:26.355962 30315 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:26.169298 30631 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.427533 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.383911 30781 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:26.424350 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:26.420686 30786 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20240426 13:21:26.387753 30768 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:26.361040 30614 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:26.448005 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:26.432335 30794 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.782s	user 0.007s	sys 0.003s
I20240426 13:21:26.356210 30817 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.077s
W20240426 13:21:26.464805 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 495595us
I20240426 13:21:26.437331 30315 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:26.456642 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.384449 30742 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20240426 13:21:26.358556 30848 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:26.358355 30718 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.470149 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.465381 30315 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.490684 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.463899 30787 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:26.446280 30792 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:26.453524 30632 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.455442 30836 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.480942 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.481654 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.482473 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.511584 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.512354 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.483285 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.391034 30680 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.484194 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.457527 30678 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:26.494740 25619 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:26.456945 30614 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:26.500661 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.501303 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:26.475466 24950 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.015s	sys 0.031s
I20240426 13:21:26.477077 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
W20240426 13:21:26.664052 24950 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.015s	sys 0.031s
I20240426 13:21:26.484258 30681 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20240426 13:21:26.505229 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2197) took 1369 ms. Trace:
I20240426 13:21:26.507639 25618 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:26.508620 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.514225 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.541236 30737 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
W20240426 13:21:26.561319 30864 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20240426 13:21:26.611413 30315 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.493095 30591 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:26.539695 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.536250 30201 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.500828 30774 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:26.536911 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.600562 25624 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:26.476481 30776 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20240426 13:21:26.563010 30448 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:26.544559 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:26.548516 30858 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20240426 13:21:26.699100 30759 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:26.564705 30704 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:26.612738 30409 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:26.550558 24956 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:21:26.730433 30794 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.560084 30760 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:26.560523 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3003) took 1463 ms. Trace:
I20240426 13:21:26.612370 30836 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.605007 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.558961 30850 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.497010 30690 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.723691 30837 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
W20240426 13:21:26.567755 30716 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.659s	user 0.002s	sys 0.013s
I20240426 13:21:26.553081 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.766737 25615 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:26.766804 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2198) took 1631 ms. Trace:
I20240426 13:21:26.545591 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2555) took 2900 ms. Trace:
I20240426 13:21:26.612308 30848 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:26.562517 25100 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:26.539398 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.767532 25030 rpcz_store.cc:276] 0426 13:21:25.135709 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.396745 (+261036us) service_pool.cc:225] Handling call
0426 13:21:25.412056 (+ 15311us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.425564 (+ 13508us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.766640 (+1341076us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b54001059d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:26.766755 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34399104}
I20240426 13:21:26.550647 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:26.540551 30861 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20240426 13:21:26.541042 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.693835 24947 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:26.703723 25619 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:26.723825 25026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.008s	sys 0.039s
W20240426 13:21:26.771447 25026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.008s	sys 0.040s
I20240426 13:21:26.504262 30672 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.162s
W20240426 13:21:26.727381 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.017s	sys 0.041s
I20240426 13:21:26.692482 25023 rpcz_store.cc:276] 0426 13:21:25.135459 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.396422 (+260963us) service_pool.cc:225] Handling call
0426 13:21:25.477313 (+ 80891us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.477555 (+   242us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:26.076496 (+598941us) log.cc:842] Serialized 14 byte log entry
0426 13:21:26.504711 (+428215us) log.cc:842] Serialized 14 byte log entry
0426 13:21:26.504866 (+   155us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:26.504920 (+    54us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:26.504974 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.505050 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:26.505168 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":468066,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":1728,"raft.queue_time_us":41680,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":116796,"spinlock_wait_cycles":101970688,"thread_start_us":912596,"threads_started":2}
I20240426 13:21:26.662776 30768 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:26.727098 30799 consensus_meta.cc:287] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.607s	user 0.016s	sys 0.000s
I20240426 13:21:26.540411 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:26.749747 30253 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.699770 30737 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:26.504671 30730 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.772725 30716 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.790426 30730 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.792176 30716 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.792418 30730 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.793035 25025 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:26.726310 30591 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:26.756609 25024 rpcz_store.cc:276] 0426 13:21:25.097602 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.116841 (+ 19239us) service_pool.cc:225] Handling call
0426 13:21:25.117123 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.897621 (+780498us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.560337 (+662716us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b54004155d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:26.560476 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1793364224}
I20240426 13:21:26.735301 30845 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.504499 30691 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.701305 30650 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.148s
I20240426 13:21:26.762368 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.799747 24951 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:26.698809 30721 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20240426 13:21:26.496650 30811 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:26.767730 25098 rpcz_store.cc:276] 0426 13:21:23.645381 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.007066 (+1361685us) service_pool.cc:225] Handling call
0426 13:21:25.007355 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:25.416145 (+408790us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.545357 (+1129212us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b5400169754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:26.545533 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":883766016}
W20240426 13:21:26.726830 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 187248us
I20240426 13:21:26.771406 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:26.772560 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.017s	sys 0.041s
I20240426 13:21:26.768994 24955 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:26.801725 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.803015 24955 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:26.803198 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.703706 30742 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:26.780586 30863 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20240426 13:21:26.780925 30707 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:26.784834 30201 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:26.785426 30678 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:26.785702 30680 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.766686 30690 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.764477 30836 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.787521 30315 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.759315 30760 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:26.787838 30850 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:26.788508 30768 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.789247 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2579) took 1040 ms. Trace:
I20240426 13:21:26.796454 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.825525 25097 rpcz_store.cc:276] 0426 13:21:25.748430 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.750871 (+  2441us) service_pool.cc:225] Handling call
0426 13:21:25.751198 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:25.755531 (+  4333us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.755569 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:25.760525 (+  4956us) log.cc:842] Serialized 26 byte log entry
0426 13:21:26.073238 (+312713us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:26.073275 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.073335 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:26.789069 (+715734us) raft_consensus.cc:1671] finished
0426 13:21:26.789089 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:26.789197 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16496,"raft.queue_time_us":7513,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":10084,"thread_start_us":268676,"threads_started":1,"wal-append.queue_time_us":577723}
I20240426 13:21:26.754078 30774 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:26.761638 30816 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:26.774372 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:26.809728 24955 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:26.809132 30811 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.815550 24947 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:26.816427 25025 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:26.836171 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:26.817157 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:26.816473 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.815438 30794 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.818724 30704 raft_consensus.cc:2741] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20240426 13:21:26.815519 30807 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:26.816157 30786 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:26.798254 30718 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.837782 24953 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:26.840301 24955 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20240426 13:21:26.840970 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.823267 30678 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:26.819234 30776 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.819840 30707 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:26.849480 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:26.874652 25022 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.850597 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.827837 30680 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.875289 25031 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:26.838395 30811 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:26.849241 30794 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.854557 30836 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.854943 30768 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.856056 24952 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:26.854585 25024 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:26.857098 30642 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:26.857534 30705 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.613s	user 0.006s	sys 0.001s
I20240426 13:21:26.860586 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.827957 30793 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.036s
I20240426 13:21:26.819501 30742 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.842164 30591 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.863195 30714 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:26.863603 30603 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:21:26.863788 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47417us
I20240426 13:21:26.897272 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.898587 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.899082 30774 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.900077 30807 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:26.868860 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:26.872123 24955 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:26.900712 30807 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:26.900976 30840 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:26.901118 30807 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:26.901631 30807 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:26.903349 30870 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:26.877836 30787 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:26.905251 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.906253 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.870668 24950 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:26.815449 30845 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:26.873520 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 373736us
I20240426 13:21:26.879238 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.879818 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:26.880753 25031 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
W20240426 13:21:26.872339 30867 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20240426 13:21:26.871642 30844 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:26.892247 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 155632us
I20240426 13:21:26.876817 30448 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.825680 30631 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.908221 25097 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:26.919549 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.917023 25022 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:26.920231 25099 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:26.872570 30786 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.921967 25031 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:26.917825 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.918618 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.918869 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.927923 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.928524 25101 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:26.928741 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:26.931434 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.932617 24954 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:26.881561 30680 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.934780 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.936902 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.877069 30776 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:26.938621 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:26.939342 24955 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:26.914628 30440 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:26.901481 30840 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:26.941076 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:26.939044 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.944494 25619 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:26.944840 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:26.945438 25097 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.934774 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.946508 25617 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:26.921489 30642 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:26.923784 30768 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.948479 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:26.892807 30705 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.939543 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:26.986460 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1037) took 1066 ms. Trace:
I20240426 13:21:26.969246 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.986848 24949 rpcz_store.cc:276] 0426 13:21:25.920297 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.920468 (+   171us) service_pool.cc:225] Handling call
0426 13:21:25.961448 (+ 40980us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:25.979728 (+ 18280us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:25.979759 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:25.980242 (+   483us) log.cc:842] Serialized 26 byte log entry
0426 13:21:26.450483 (+470241us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:26.450538 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:26.450622 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:26.986250 (+535628us) raft_consensus.cc:1671] finished
0426 13:21:26.986284 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:26.986410 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40283,"raft.queue_time_us":18954,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":13922,"thread_start_us":469896,"threads_started":1,"wal-append.queue_time_us":780847}
I20240426 13:21:26.928923 30591 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:26.924058 30868 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:26.945214 30849 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20240426 13:21:26.975033 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:26.975466 30879 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:26.944994 30730 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:26.977635 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:26.978189 30884 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:26.979346 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:26.980022 30742 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:26.954119 25025 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:26.982256 25030 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:26.983258 30776 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:26.935746 30816 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:26.939807 30834 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:26.876780 30760 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:26.945680 30603 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
W20240426 13:21:27.024813 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 189823us
W20240426 13:21:26.942610 30778 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.047s	user 0.015s	sys 0.004s
I20240426 13:21:27.079079 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:26.923286 30448 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:27.037842 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132630us
I20240426 13:21:27.082573 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.082729 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:27.083801 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.011934 30781 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.557s	user 0.004s	sys 0.005s
I20240426 13:21:27.084656 30781 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.085026 25027 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:21:27.086661 30781 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:27.048758 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69476.4us
I20240426 13:21:27.047870 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:27.087729 24949 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:27.076858 30603 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:27.078859 30742 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.082270 30835 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:26.953836 30440 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.009732 30642 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.044564 25097 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:26.999094 30631 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:26.984805 30645 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.069869 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.069502 30774 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:26.924454 30707 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.029800 30787 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.010046 30793 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:27.030171 25104 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:26.999004 30768 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.079162 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.048259 30730 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:27.076689 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 101720us
I20240426 13:21:27.089177 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:27.093902 25099 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:21:27.073480 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:27.106282 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.113600 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:27.127059 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.128132 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.114070 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 136489us
I20240426 13:21:27.089464 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:27.105902 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.131233 25097 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:27.098901 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:27.103780 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
W20240426 13:21:27.123292 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44225.7us
I20240426 13:21:27.076736 30690 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.096848 30887 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:27.096747 30885 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:27.092664 30884 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:27.092464 30835 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.165189 30835 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:27.167985 30678 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.169234 30614 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.169945 30739 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.170486 30739 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:27.147414 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58022us
I20240426 13:21:27.136314 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.153090 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 273358us
I20240426 13:21:27.094630 30680 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.154029 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:27.183039 25023 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:27.183199 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:27.172457 25622 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:27.108251 30774 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.154958 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.091338 30603 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:27.191337 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55130us
I20240426 13:21:27.046538 30879 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.097949 30705 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.203778 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.152478 30885 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:27.150831 30887 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.100163 30873 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.116192 30793 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.116765 30704 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:27.095358 30786 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.098119 30794 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.101521 30440 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.103065 30642 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.152210 30847 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:27.161146 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.161491 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.070760 30840 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.162454 30807 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.163774 30845 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.091825 30742 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.134369 30781 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.093456 30448 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.231755 30873 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.115998 30834 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:27.292208 30834 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.119309 30849 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:27.118391 30631 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:27.192165 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.296286 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104191us
I20240426 13:21:27.297513 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.146835 30876 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:27.201081 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.307008 25022 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:27.203088 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:27.146050 30893 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:27.210232 30904 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:27.155019 30884 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:27.147264 30896 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:27.203685 30692 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:27.218055 25101 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:27.104374 30645 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.356969 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:27.224478 30895 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20240426 13:21:27.225808 30768 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.227494 30812 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.191556 30774 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:27.295404 30799 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:27.372256 25615 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:27.356796 30778 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:27.295845 30714 consensus_queue.cc:1159] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1590f80bdade41c6a755b8501c55fe16" responder_term: 17 status { last_received { term: 16 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:27.357014 30896 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.384739 30877 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:27.385236 30877 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:27.386049 30905 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20240426 13:21:27.386869 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.387107 30877 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.387522 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.388226 25101 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:27.389596 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.296443 30781 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:27.433010 30875 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:21:27.433773 30884 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.435256 30884 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:27.371630 30904 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:27.269071 24955 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:27.444980 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.445317 25617 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:27.282021 25103 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
W20240426 13:21:27.284453 30886 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20240426 13:21:27.285724 30807 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.285861 30742 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.286304 30840 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:27.462793 30903 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20240426 13:21:27.287751 30879 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.288269 30614 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.185217 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:27.464627 25623 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:27.464767 30645 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.466621 30739 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.467491 30904 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.394276 30896 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:27.297960 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2560) took 2671 ms (client timeout 2009 ms). Trace:
I20240426 13:21:27.472014 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 762) took 1396 ms. Trace:
I20240426 13:21:27.186501 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:27.472774 25622 rpcz_store.cc:276] 0426 13:21:26.075241 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.104287 (+ 29046us) service_pool.cc:225] Handling call
0426 13:21:27.471810 (+1367523us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002a58d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:27.471947 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2439738112}
W20240426 13:21:27.475471 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 291362us
W20240426 13:21:27.472630 25096 rpcz_store.cc:269] 0426 13:21:24.626634 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.637470 (+ 10836us) service_pool.cc:225] Handling call
0426 13:21:27.281658 (+2644188us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:27.297868 (+ 16210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1383481600}
I20240426 13:21:27.478219 25622 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:27.310743 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:27.282320 30863 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240426 13:21:27.313807 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 130701us
I20240426 13:21:27.273234 30776 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:27.219700 30887 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.280647 30642 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:27.293962 30847 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:27.270702 30850 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20240426 13:21:27.305974 30849 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.309652 30440 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:27.422775 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.423424 30873 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.439087 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:27.555243 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.430924 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:27.232082 30793 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:27.525242 30840 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20240426 13:21:27.498095 30845 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:27.557222 30845 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:27.558034 30845 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.270695 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.445508 24955 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:27.560869 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.561226 30835 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.561954 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.562548 24955 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:27.461599 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2209) took 1610 ms. Trace:
I20240426 13:21:27.490742 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:27.496186 30778 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20240426 13:21:27.564270 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.563207 25029 rpcz_store.cc:276] 0426 13:21:25.850825 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.850931 (+   106us) service_pool.cc:225] Handling call
0426 13:21:25.870615 (+ 19684us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:25.870939 (+   324us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:26.601796 (+730857us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b180035aee0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:21:26.984886 (+383090us) log.cc:842] Serialized 14 byte log entry
0426 13:21:27.231785 (+246899us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:27.270327 (+ 38542us) log.cc:842] Serialized 26 byte log entry
0426 13:21:27.270512 (+   185us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:27.270546 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:27.270610 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:27.446475 (+175865us) raft_consensus.cc:1671] finished
0426 13:21:27.461427 (+ 14952us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:27.461547 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1702,"prepare.queue_time_us":165044,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":102245,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":7160,"spinlock_wait_cycles":11871104,"thread_start_us":549918,"threads_started":3,"wal-append.queue_time_us":248375}
I20240426 13:21:27.565130 25104 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20240426 13:21:27.563447 30793 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.566581 25097 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:27.497354 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 186666us
I20240426 13:21:27.567281 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.568464 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:27.569293 30614 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:27.569810 30614 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:27.571463 30840 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.571908 30840 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.572706 30614 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:27.308662 30885 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:27.573812 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 142927us
I20240426 13:21:27.574625 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:27.509824 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.575592 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.575769 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:27.526221 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.576180 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.576575 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.576822 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:27.576969 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:27.577145 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.577529 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:27.549048 25101 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:27.550526 30440 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:27.553673 30896 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.573155 30614 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:27.573870 30840 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.565980 24953 raft_consensus.cc:2396] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20240426 13:21:27.579119 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 308626us
I20240426 13:21:27.630946 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.631390 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.632265 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:27.633284 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.374027 30799 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.635843 30799 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.573670 30885 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:27.489182 30668 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:27.636936 30668 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:27.637670 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:27.637866 25105 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:27.638280 24955 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:27.638770 30668 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:27.473301 24854 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 15, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 15 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:27.594107 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.640193 24952 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:27.592492 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.642086 25024 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:27.642886 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.643409 25024 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:27.596004 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.644111 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.644925 25029 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.644151 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2176) took 2520 ms. Trace:
I20240426 13:21:27.639053 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:27.646760 25103 rpcz_store.cc:276] 0426 13:21:25.124004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.511571 (+387567us) service_pool.cc:225] Handling call
0426 13:21:27.281819 (+1770248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:27.455266 (+173447us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:27.643914 (+188648us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54000eb350. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:27.644086 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4070349568}
I20240426 13:21:27.647513 30859 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:27.652459 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.650771 25103 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:27.638921 25105 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:27.549418 30849 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:27.655295 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.650207 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:27.579555 30893 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.576217 30879 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:21:27.625398 30811 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.739s	user 0.010s	sys 0.008s
I20240426 13:21:27.657393 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.511557 30739 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.119s
I20240426 13:21:27.546967 30642 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.659772 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.593470 30835 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.669945 24952 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:27.595197 30798 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:27.659528 25027 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:27.672196 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:27.673310 25029 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20240426 13:21:27.730705 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58584us
W20240426 13:21:27.730674 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73383.7us
I20240426 13:21:27.732087 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.732513 25103 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:27.733028 25103 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:27.737942 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.739526 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.740082 25096 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:27.733762 25103 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.741288 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:27.741616 25103 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:27.741516 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.661682 30631 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:21:27.675415 30815 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.871s	user 0.027s	sys 0.041s
I20240426 13:21:27.749598 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.676776 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:27.750926 25105 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:27.741896 25104 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.670985 30879 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.671471 30496 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:27.659190 30915 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20240426 13:21:27.751715 25104 raft_consensus.cc:603] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:27.736414 30604 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:27.747465 30914 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20240426 13:21:27.754284 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77551.7us
I20240426 13:21:27.671635 30906 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:27.754861 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.746985 30907 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20240426 13:21:27.758447 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:27.775326 25105 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:27.752887 25103 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:27.742507 30448 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:27.763432 30793 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.746371 30889 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:27.742970 30614 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.746207 30798 raft_consensus.cc:2741] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20240426 13:21:27.764607 30781 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.761143 30604 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:27.765396 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2355) took 2011 ms (client timeout 2672 ms). Trace:
I20240426 13:21:27.769063 30642 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.761394 30742 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20240426 13:21:27.770226 24851 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 280) took 52247 ms (client timeout 15000 ms). Trace:
I20240426 13:21:27.770566 30668 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:27.772980 30912 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:27.772588 24854 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 10, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:27.773391 24950 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:27.773983 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:27.750218 30815 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.952s	user 0.030s	sys 0.050s
I20240426 13:21:27.776590 25103 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 7
I20240426 13:21:27.659036 30859 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.756628 30879 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:27.676654 30811 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.745738 30918 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20240426 13:21:27.795946 24948 rpcz_store.cc:269] 0426 13:21:25.753993 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:25.754250 (+   257us) service_pool.cc:225] Handling call
0426 13:21:27.765261 (+2011011us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:27.765347 (+    86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961203456}
I20240426 13:21:27.796525 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:27.774274 30799 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.804579 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:27.851279 25104 raft_consensus.cc:595] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.851605 25104 raft_consensus.cc:603] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:21:27.788471 30718 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.005s	sys 0.005s
I20240426 13:21:27.782039 30876 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:27.798529 30834 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.671286 30893 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:27.800261 24851 rpcz_store.cc:269] 0426 13:20:35.522861 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:35.523270 (+   409us) service_pool.cc:225] Handling call
0426 13:21:27.770165 (+52246895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:20.196374 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:20.199891 (+  3517us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:20.199946 (+    55us) write_op.cc:427] Acquired schema lock
0426 13:21:20.199966 (+    20us) tablet.cc:660] Decoding operations
0426 13:21:20.201119 (+  1153us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:20.201196 (+    77us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:20.201231 (+    35us) tablet.cc:683] Acquiring locks for 12 operations
0426 13:21:20.202342 (+  1111us) tablet.cc:699] Row locks acquired
0426 13:21:20.202374 (+    32us) write_op.cc:260] PREPARE: finished
0426 13:21:20.202558 (+   184us) write_op.cc:270] Start()
0426 13:21:20.202687 (+   129us) write_op.cc:275] Timestamp: P: 1714137680202481 usec, L: 0
0426 13:21:20.202722 (+    35us) op_driver.cc:331] REPLICATION: starting
0426 13:21:20.203377 (+   655us) log.cc:842] Serialized 4719 byte log entry
0426 13:21:20.627014 (+423637us) op_driver.cc:444] REPLICATION: finished
0426 13:21:20.856489 (+229475us) write_op.cc:300] APPLY: starting
0426 13:21:20.856611 (+   122us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:21.781766 (+925155us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:21.781894 (+   128us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:22.327077 (+545183us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:22.393388 (+ 66311us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0426 13:21:22.399829 (+  6441us) write_op.cc:311] APPLY: finished
0426 13:21:22.484856 (+ 85027us) log.cc:842] Serialized 113 byte log entry
0426 13:21:22.485051 (+   195us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:22.485184 (+   133us) write_op.cc:446] Released schema lock
0426 13:21:22.485241 (+    57us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:22.485573 (+   332us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:22.485606 (+    33us) write_op.cc:446] Released schema lock
0426 13:21:22.570098 (+ 84492us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b1800366dc0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":383179,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":218135,"apply.run_cpu_time_us":25902,"apply.run_wall_time_us":1629557,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54311,"num_ops":12,"prepare.queue_time_us":284484,"prepare.run_cpu_time_us":9223,"prepare.run_wall_time_us":163356,"raft.queue_time_us":218439,"raft.run_cpu_time_us":9472,"raft.run_wall_time_us":101989,"replication_time_us":424180,"spinlock_wait_cycles":2842880,"thread_start_us":419744,"threads_started":4,"wal-append.queue_time_us":205243,"wal-append.run_cpu_time_us":28827,"wal-append.run_wall_time_us":2190002}]]}
I20240426 13:21:27.853469 30918 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:27.843591 25103 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:27.843638 24854 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:27.861207 25103 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:27.758155 30496 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.797704 30604 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:27.658862 30847 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:27.859566 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85657.5us
I20240426 13:21:27.851668 24948 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20240426 13:21:27.853392 30718 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s	user 0.005s	sys 0.005s
I20240426 13:21:27.869740 30913 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:27.870163 30913 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.870488 30913 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:27.872699 30923 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20240426 13:21:27.797792 30448 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20240426 13:21:27.874764 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:27.854609 30614 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:27.807976 30793 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20240426 13:21:27.858806 30876 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:27.886665 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:27.795871 30906 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20240426 13:21:27.904294 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:27.904973 25101 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.906160 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:27.808362 30911 consensus_queue.cc:1159] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 15 status { last_received { term: 13 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:27.888722 30556 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:27.893366 30912 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:27.873787 30681 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:27.916960 30859 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.917145 30681 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.917529 30681 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.897423 30603 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.901439 30876 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.921682 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.857134 30799 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.864544 30918 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:27.924206 30799 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.925020 30496 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.864989 30834 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.926911 30834 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.932201 30665 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.932713 30665 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.933887 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.934720 30665 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.907732 25104 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:27.907258 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.935719 30839 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 2.138s	user 0.002s	sys 0.006s
I20240426 13:21:27.906507 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:27.936990 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:27.913039 30930 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:27.884217 30642 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:27.938899 30930 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.939285 30930 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:27.939045 30642 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.941076 30642 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.941582 30642 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:27.881661 30896 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.934669 25098 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:27.943869 25622 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
W20240426 13:21:27.904320 30920 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20240426 13:21:27.848214 30798 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20240426 13:21:27.855983 30705 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20240426 13:21:27.938081 30811 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.947094 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:27.948177 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2888) took 1467 ms. Trace:
I20240426 13:21:27.948591 24951 rpcz_store.cc:276] 0426 13:21:26.480314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.800362 (+320048us) service_pool.cc:225] Handling call
0426 13:21:26.800720 (+   358us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:26.833365 (+ 32645us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:26.833392 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:26.833434 (+    42us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:26.833459 (+    25us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:27.947927 (+1114468us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b54000bf254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:27.947979 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:27.948118 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74682624}
I20240426 13:21:27.951352 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:27.951875 25101 raft_consensus.cc:595] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.947890 30849 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.956848 30876 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:27.959517 30774 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.585s	user 0.003s	sys 0.003s
I20240426 13:21:27.952386 30879 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:27.961339 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3031) took 1129 ms. Trace:
I20240426 13:21:27.948954 30859 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:27.961923 30859 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:27.961704 25025 rpcz_store.cc:276] 0426 13:21:26.831548 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.840704 (+  9156us) service_pool.cc:225] Handling call
0426 13:21:27.961198 (+1120494us) spinlock_profiling.cc:243] Waited 86 ms on lock 0x7b54004173d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:27.961288 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197836544}
I20240426 13:21:27.962460 30859 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.964725 30879 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.955983 30849 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.937793 30681 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.965708 30870 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240426 13:21:27.966260 30884 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.531s	user 0.007s	sys 0.000s
I20240426 13:21:27.966835 24954 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:27.972622 30730 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:27.958518 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:27.973803 25101 raft_consensus.cc:595] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:27.974694 30874 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:27.975070 30874 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:27.976435 30691 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:27.976789 30691 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:27.977520 30691 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:27.978538 30876 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.978973 30681 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:27.887111 24854 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 13, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 13 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:27.975212 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
W20240426 13:21:27.980415 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43499.4us
I20240426 13:21:27.981876 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:27.982672 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:27.992911 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:27.993484 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84700.1us
I20240426 13:21:27.996337 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:27.997025 25025 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:27.998927 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:28.001866 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77182.9us
I20240426 13:21:28.002056 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.006637 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.009598 25101 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:28.009008 24854 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.030948 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.031682 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.032498 25096 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.029582 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.001245 30692 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:28.036005 30692 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:27.992831 30404 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:28.007686 30315 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:28.036422 30692 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.007575 30730 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.033s
I20240426 13:21:28.041214 30730 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:21:28.029901 30737 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.042757 30737 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:27.975777 30874 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:28.044166 30768 consensus_queue.cc:1159] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 15 status { last_received { term: 15 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 15 index: 11 } }
I20240426 13:21:28.044622 30853 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:28.045018 30874 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:28.045177 30853 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.047281 30853 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:27.959961 30774 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.049736 25026 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:28.049991 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.050712 30774 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.038573 30404 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:28.052670 30404 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.053946 30404 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.024430 30884 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:28.060541 30884 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.055076 30930 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.061419 30884 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.065009 30604 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.068058 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.068185 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:28.068867 30721 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:28.070268 30670 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:28.034786 30913 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.071401 25104 raft_consensus.cc:2354] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:28.071563 25025 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 15
I20240426 13:21:28.071676 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1041) took 1727 ms. Trace:
I20240426 13:21:28.072230 30913 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.072183 24956 rpcz_store.cc:276] 0426 13:21:26.344689 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.344849 (+   160us) service_pool.cc:225] Handling call
0426 13:21:26.550284 (+205435us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:26.752409 (+202125us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:28.071499 (+1319090us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400366ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.071623 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434648320}
I20240426 13:21:28.073177 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.073890 25104 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:28.074569 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.075717 24996 leader_election.cc:398] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:28.076017 24996 leader_election.cc:401] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:28.077143 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:28.077340 25104 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.075413 30918 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.029340 24854 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 14, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.078174 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.084901 25025 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:28.089323 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:28.089121 25101 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:28.094009 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.094132 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.098707 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:28.093549 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.101095 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.101496 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.101534 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:28.102509 25025 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.099920 24854 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 17, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 17 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.104152 24854 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 17, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 17 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.101868 25616 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:28.105054 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:28.105024 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.107512 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.106484 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.108696 25031 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20240426 13:21:28.108892 25101 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:28.105002 25096 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:28.080377 30496 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:28.112738 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.114356 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:28.112661 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.110921 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:28.115305 25028 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:28.068987 30604 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.118110 24854 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 12 to 15, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.090620 30930 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.110491 30681 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:28.139189 30681 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:28.140379 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.110363 30868 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:28.140976 25623 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:28.141310 30868 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:28.127008 30934 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:28.128419 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:28.132685 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:28.133590 30918 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.150419 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:28.150458 30868 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.152794 25621 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:28.040249 30859 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.134088 30691 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.134716 30879 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:28.136863 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:28.110797 30876 raft_consensus.cc:2741] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20240426 13:21:28.140051 30681 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.124074 30692 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:28.143124 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 289141us
I20240426 13:21:28.172178 24854 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.144203 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.146226 25098 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:28.146363 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:28.143529 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.147364 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.179046 25617 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:28.179910 24949 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20240426 13:21:28.180372 30883 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:28.180948 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.147665 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.181635 24949 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:28.181861 24956 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:28.182704 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.182817 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.183636 24956 raft_consensus.cc:2354] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:28.145797 30934 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:28.185859 30934 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.186956 30934 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:28.149765 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.156976 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.189836 25022 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:28.157979 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.159456 24948 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:28.192445 24947 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:28.160017 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.175565 30315 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.194926 30315 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:28.196686 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53233.2us
I20240426 13:21:28.163233 25616 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:28.199187 30645 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:28.200323 30850 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:28.176498 30668 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.201074 30668 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.201747 24854 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 15, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 15 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.202934 30668 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.203761 24854 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 17. New cstate: current_term: 17 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.153860 25025 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:28.171337 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:28.175254 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.177574 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.139701 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:28.205842 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2174) took 4390 ms (client timeout 2137 ms). Trace:
W20240426 13:21:28.206964 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62848.6us
W20240426 13:21:28.217909 25102 rpcz_store.cc:269] 0426 13:21:23.815023 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:24.582422 (+767399us) service_pool.cc:225] Handling call
0426 13:21:28.205652 (+3623230us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.205768 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4316803456}
I20240426 13:21:28.218299 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.207832 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:28.183633 24949 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:28.220108 24955 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:28.219588 25624 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:28.164501 25101 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:28.211146 25623 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:28.212359 24950 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:21:28.210875 25099 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20240426 13:21:28.226723 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49211us
I20240426 13:21:28.228018 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.228394 24951 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:28.229722 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.231379 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:28.233755 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2912) took 1080 ms. Trace:
I20240426 13:21:28.234264 24954 rpcz_store.cc:276] 0426 13:21:27.153440 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.386655 (+233215us) service_pool.cc:225] Handling call
0426 13:21:28.233567 (+846912us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54002cbfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.233688 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332239616}
I20240426 13:21:28.237535 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.242985 25104 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:28.252521 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.276764 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.276578 24854 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 16, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 16 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:28.278086 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:28.279749 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51788.5us
I20240426 13:21:28.281297 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.304354 25619 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:28.278842 25098 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:28.244359 30556 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:28.311156 30603 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:28.235260 30843 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:28.206492 30879 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:28.239310 30730 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:28.311847 30937 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:28.312394 30937 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:28.312846 30937 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:28.314502 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 143233us
I20240426 13:21:28.316709 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:28.317859 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65429.7us
I20240426 13:21:28.318264 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.318583 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:28.319861 30928 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:28.320225 30665 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:28.320742 30665 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.320901 30591 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:28.321444 30591 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:28.321940 30591 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:28.321560 30665 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.224681 30837 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:28.219599 30721 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:28.324999 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 87546.1us
I20240426 13:21:28.324952 30721 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.325609 30642 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:28.325857 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:28.326449 30906 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:28.326828 30906 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:28.327172 30906 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:28.327265 30603 raft_consensus.cc:979] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:28.239632 30817 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:28.328092 30730 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:28.328683 30730 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.329288 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.326013 30642 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:28.330376 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.330899 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:28.334770 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.335345 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.356649 25104 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:28.339259 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:28.360298 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.360591 24854 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.363258 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:28.362614 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 778) took 1545 ms (client timeout 1866 ms). Trace:
I20240426 13:21:28.363889 25022 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
W20240426 13:21:28.364603 25030 rpcz_store.cc:269] 0426 13:21:26.816813 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.816917 (+   104us) service_pool.cc:225] Handling call
0426 13:21:28.362433 (+1545516us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.362543 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333513600}
I20240426 13:21:28.367058 24950 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20240426 13:21:28.367547 24950 pending_rounds.cc:76] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:21:28.367800 24950 pending_rounds.cc:97] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20240426 13:21:28.368093 24950 raft_consensus.cc:2866] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:28.360872 30690 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:28.420353 30690 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.422420 30690 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.423120 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.423871 25624 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:28.424639 30692 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:28.425864 30940 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20240426 13:21:28.426260 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.427004 25623 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:28.225328 30645 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.132s
W20240426 13:21:28.427644 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72973.6us
I20240426 13:21:28.344192 30793 raft_consensus.cc:1067] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:28.344671 30883 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20240426 13:21:28.359577 30642 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:28.456174 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 116992us
I20240426 13:21:28.456974 24854 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 12, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 12 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.344662 30448 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:28.325256 30678 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.450353 30730 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.516803 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 777) took 1706 ms. Trace:
I20240426 13:21:28.517416 25023 rpcz_store.cc:276] 0426 13:21:26.810081 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.810185 (+   104us) service_pool.cc:225] Handling call
0426 13:21:26.826058 (+ 15873us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:27.188860 (+362802us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:28.516482 (+1327622us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:28.516731 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":797833472}
I20240426 13:21:28.329509 30601 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.123s
W20240426 13:21:28.498883 30942 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20240426 13:21:28.312623 30879 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.525691 24854 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 8, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.548703 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:28.543108 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50064.6us
I20240426 13:21:28.499475 30798 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:28.585126 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:28.586687 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:28.588224 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45922.1us
I20240426 13:21:28.590600 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:28.592725 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:28.608686 30946 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20240426 13:21:28.610057 30835 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:28.610438 30835 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:28.610755 30835 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:28.610834 30928 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:28.611018 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.611399 30928 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.612449 25104 raft_consensus.cc:1203] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 15.8->[16.9-18.10]   Dedup: 16.9->[18.10-18.10]
I20240426 13:21:28.612947 25104 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: [16.9-18.10]
I20240426 13:21:28.601558 30845 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20240426 13:21:28.598033 30913 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:28.618369 30835 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.620270 30721 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.620493 30835 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.621860 30835 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:28.622185 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.622637 30835 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:21:28.622766 25099 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:28.623272 30928 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.624456 30937 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.626369 30937 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.628494 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.628880 30859 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.632218 30915 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:28.632592 30915 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:28.633332 30915 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.634227 30915 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.637089 30591 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.639083 30591 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.639860 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 999) took 1381 ms. Trace:
I20240426 13:21:28.640556 25105 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:28.641211 30760 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:28.640233 25100 rpcz_store.cc:276] 0426 13:21:27.258239 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.258397 (+   158us) service_pool.cc:225] Handling call
0426 13:21:27.262474 (+  4077us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:27.357735 (+ 95261us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:27.357779 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:27.357830 (+    51us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:27.357872 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:28.639648 (+1281776us) spinlock_profiling.cc:243] Waited 95 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:28.639700 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:28.639810 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218554240}
I20240426 13:21:28.587641 30917 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:28.644681 30917 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:28.645017 30917 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:28.645480 30707 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.645527 30906 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.627763 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.648885 24947 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:28.588789 30678 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:28.649559 30678 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.649693 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1173) took 1429 ms. Trace:
I20240426 13:21:28.650307 30678 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.651661 30678 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:28.524855 30448 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.653945 30448 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.600916 30885 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:28.655407 30885 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.653999 30906 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:28.657263 30885 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.658715 30924 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.645393 30833 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:28.659225 30833 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:28.659206 30924 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.657974 25099 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:28.660019 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.660877 24951 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:28.660421 30833 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.661692 30924 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.661808 30833 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:28.659348 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:28.668666 25100 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20240426 13:21:28.671265 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 771) took 1732 ms (client timeout 1957 ms). Trace:
W20240426 13:21:28.671689 25615 rpcz_store.cc:269] 0426 13:21:26.938546 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:26.939335 (+   789us) service_pool.cc:225] Handling call
0426 13:21:28.671121 (+1731786us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.671213 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":889194240}
W20240426 13:21:28.673278 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2914) took 1515 ms (client timeout 1639 ms). Trace:
W20240426 13:21:28.673668 24952 rpcz_store.cc:269] 0426 13:21:27.157715 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.294599 (+136884us) service_pool.cc:225] Handling call
0426 13:21:28.673104 (+1378505us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b540004b354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.673223 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":649543552}
I20240426 13:21:28.674679 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.674896 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.675269 25098 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:28.675464 25105 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:28.676168 24953 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:28.676244 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:28.677423 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2379) took 1702 ms. Trace:
I20240426 13:21:28.677778 24953 rpcz_store.cc:276] 0426 13:21:26.974533 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.649984 (+675451us) service_pool.cc:225] Handling call
0426 13:21:28.675974 (+1025990us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:28.676724 (+   750us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:28.677262 (+   538us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540004b350. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.677373 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2358962560}
I20240426 13:21:28.678694 24953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:28.679281 24953 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:28.679661 24953 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:28.681839 30930 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:28.682744 30930 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
W20240426 13:21:28.683892 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.866s (SENT) [suppressed 5 similar messages]
I20240426 13:21:28.688868 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.683633 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:28.650029 25619 rpcz_store.cc:276] 0426 13:21:27.220033 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.220226 (+   193us) service_pool.cc:225] Handling call
0426 13:21:27.220501 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:28.310847 (+1090346us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:28.649482 (+338635us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540058e654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:28.649643 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2433976192}
I20240426 13:21:28.658962 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.685173 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.695531 25099 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:28.698833 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:28.699563 25619 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:28.701648 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:28.704746 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.705725 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:28.706233 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.685632 24947 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:28.707557 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.720933 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:28.726140 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.727308 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:28.731463 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:28.732110 25619 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:28.684422 30793 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:28.760121 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:28.746618 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:28.760771 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 780) took 1682 ms (client timeout 1960 ms). Trace:
I20240426 13:21:28.687091 30603 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:28.686092 30930 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.721207 30760 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:21:28.786998 30929 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:28.718884 30877 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:28.719940 30906 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:28.736732 30937 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:28.736879 30928 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:28.789017 30798 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:28.724560 30896 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:28.788441 30859 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:28.736995 30924 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:28.759027 30591 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.765553 30705 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.799221 30928 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:28.797420 30760 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:28.804564 30798 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:28.805869 30924 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:28.810740 30896 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:28.814299 30928 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:28.829336 30896 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:28.830595 30934 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.643s	user 0.013s	sys 0.005s
I20240426 13:21:28.809590 30930 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.832815 30930 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.818460 30924 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.835960 30917 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.838704 30917 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.844014 30928 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.845969 30928 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.847534 30924 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.876921 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:28.880872 24949 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:28.955034 30924 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.876780 24854 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 9, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.731222 24953 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.955946 30690 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:28.956871 30690 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.958048 30315 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:28.958195 24953 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:28.957783 24854 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 6 to 13, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 13 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:28.884462 24950 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20240426 13:21:28.884209 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:28.884788 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:28.765539 25026 rpcz_store.cc:269] 0426 13:21:27.078423 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.078555 (+   132us) service_pool.cc:225] Handling call
0426 13:21:28.760598 (+1682043us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:28.760720 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2229581184}
I20240426 13:21:28.906101 30665 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:28.907512 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.766103 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:28.913883 30670 raft_consensus.cc:979] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:28.915563 30760 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:28.920610 30591 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.922521 30705 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:28.924546 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:28.818159 30798 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:28.928074 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 244043us
I20240426 13:21:28.959730 30690 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:28.983500 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 107078us
I20240426 13:21:28.984452 30843 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:28.984513 30440 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:28.993188 30848 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:29.004704 24954 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:28.999514 30665 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.009881 30591 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.010589 30705 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.017019 30843 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:28.971649 30642 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.024945 24854 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:29.026911 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.027961 24948 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:29.031060 30848 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:29.031992 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 265854us
W20240426 13:21:29.032747 30966 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20240426 13:21:29.013957 30440 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.033823 30440 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.038231 30665 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.041646 30631 raft_consensus.cc:1067] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:29.044802 30893 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.022065 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.052851 24947 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:29.053306 30947 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.449s	user 0.003s	sys 0.000s
I20240426 13:21:29.055961 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:29.056771 24948 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:29.057207 24948 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:29.060272 30816 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:29.060166 30404 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20240426 13:21:29.063905 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 719170us
I20240426 13:21:29.064718 30604 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:29.065224 30604 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:29.065521 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 733113us
W20240426 13:21:29.066927 30962 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20240426 13:21:29.066618 30604 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.090849 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.026971 24854 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 5 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:29.095021 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.095896 25025 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:29.096959 25104 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20240426 13:21:29.097090 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.097729 25025 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:29.108000 24955 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:29.118434 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:29.118561 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53673.7us
I20240426 13:21:29.098552 30665 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:29.127532 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71888.2us
I20240426 13:21:29.127697 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:29.134271 25100 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:29.145499 25104 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:29.149351 30716 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
W20240426 13:21:29.149618 30496 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.006s	user 0.014s	sys 0.014s
I20240426 13:21:29.149950 30716 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:29.150173 30496 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:29.150696 30798 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.152084 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.152833 30680 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:29.152940 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84199us
I20240426 13:21:29.153605 30680 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.154040 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.155081 30812 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:29.155652 30812 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:29.157426 30894 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:29.157858 30894 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.158200 30894 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.159372 30876 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:29.159852 30876 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.160261 30876 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:29.161835 30707 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.516s	user 0.007s	sys 0.014s
I20240426 13:21:29.162657 30707 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:29.164078 30670 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:29.164422 25028 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:29.164479 30670 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:29.165007 25028 pending_rounds.cc:76] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 7
I20240426 13:21:29.165367 25028 pending_rounds.cc:97] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:29.165642 25028 raft_consensus.cc:2866] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:29.166149 30848 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.165884 30670 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.167110 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.119205 30816 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20240426 13:21:29.170791 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.171710 30593 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.172163 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.178247 25030 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20240426 13:21:29.179839 30631 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.180162 30681 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.180517 30631 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.180996 30887 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.181782 30887 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.182905 30665 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.185348 30665 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.185595 30718 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.186232 30718 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.186610 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.186918 30853 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.187721 30853 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.188603 30908 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.189172 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.189339 30908 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.189903 30853 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.165884 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:29.190690 25105 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:29.190984 25105 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:29.191275 30631 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.191350 30908 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.191886 25105 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.188475 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.193085 25105 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.173673 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.197033 25031 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.180999 30681 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.198675 25022 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:29.202432 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.183933 30887 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.151298 30716 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.155972 30680 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:29.213383 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68302.8us
I20240426 13:21:29.210619 25028 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.228798 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.232512 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.172255 30593 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.235375 30798 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.239833 30812 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.241973 30440 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.244311 30440 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.245241 30631 raft_consensus.cc:979] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:29.245771 30631 raft_consensus.cc:1067] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:29.237349 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.247324 30670 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.238473 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:29.246018 25103 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:21:29.223711 30811 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.233940 30642 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.256736 30602 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:29.262217 30602 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.266315 30602 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:29.268657 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78272.7us
I20240426 13:21:29.269367 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.271032 24953 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:29.271812 30591 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:29.272138 30556 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:21:29.273002 24955 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:29.273823 30556 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:29.274341 30556 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:29.275291 30556 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.271831 30721 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.279420 30721 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.281598 30721 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.272536 30591 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:29.283263 30591 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.272202 24947 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:29.289156 25025 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:29.257599 30718 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.308841 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.318428 30776 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.319074 30776 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.320835 30681 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.314532 25028 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:29.322132 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:29.315026 24948 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.326261 24955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:29.326872 24955 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:29.327275 24955 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.325872 25100 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.331647 25099 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:29.328682 30614 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:29.333374 24949 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:29.323570 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.324477 30665 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.359210 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.358839 24948 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.298570 30811 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.376823 30847 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.376996 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.380079 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:29.381280 30404 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.396098 30404 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.398003 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:29.398264 30404 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.382647 30834 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:29.382741 30870 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:29.400278 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.401568 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.401948 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:29.402107 25098 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:29.402940 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.403292 25102 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:29.404322 30893 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.404521 25102 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20240426 13:21:29.404867 30893 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.407584 25102 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:29.407800 30812 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:29.376952 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:29.408329 30847 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.408455 30812 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.410292 30847 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.410480 30812 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.410985 30812 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.386287 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:29.387183 30971 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20240426 13:21:29.395370 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:29.398746 25616 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.400058 25024 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:29.384768 30843 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.408056 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.424427 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:29.425129 25099 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:29.427876 25099 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:29.418540 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3052) took 1107 ms. Trace:
I20240426 13:21:29.430894 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.432183 25022 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:29.433508 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:29.434687 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 226258us
I20240426 13:21:29.433321 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.436950 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.438066 25028 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.436985 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.438757 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.440861 25031 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.435514 25099 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.436882 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:29.439765 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.508491 25622 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:29.443961 25102 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:29.526392 25099 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:29.528347 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2626) took 1531 ms. Trace:
I20240426 13:21:29.528750 25103 rpcz_store.cc:276] 0426 13:21:27.997198 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.998172 (+   974us) service_pool.cc:225] Handling call
0426 13:21:27.998455 (+   283us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:29.251262 (+1252807us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:29.528045 (+276783us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003e0dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:29.528275 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2867832192}
W20240426 13:21:29.529887 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95040us
I20240426 13:21:29.523195 25025 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:29.532351 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2624) took 1535 ms. Trace:
I20240426 13:21:29.532681 30812 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.526101 25623 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:29.539211 25105 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:29.436676 30893 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.429064 30665 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:29.539613 30909 consensus_queue.cc:1159] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 20 status { last_received { term: 19 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:29.525723 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.540707 25029 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:29.541348 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.542110 30843 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.544220 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:29.544905 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:29.544989 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.545722 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.547533 30977 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:29.548915 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.544867 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.550207 24955 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.553306 25100 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:29.565446 25031 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.567471 25028 raft_consensus.cc:2362] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18. Re-sending same reply.
I20240426 13:21:29.576068 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.576938 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.584101 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.586652 24955 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.582329 25026 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:29.582669 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.593194 24954 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:29.594828 30874 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:29.595233 30874 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:29.595552 30807 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:29.547993 30602 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.597299 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.591990 24950 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:29.604365 25029 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:29.553447 25097 rpcz_store.cc:276] 0426 13:21:27.996697 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:27.997702 (+  1005us) service_pool.cc:225] Handling call
0426 13:21:28.021752 (+ 24050us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:28.022284 (+   532us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:28.022327 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:28.022911 (+   584us) log.cc:842] Serialized 26 byte log entry
0426 13:21:28.979096 (+956185us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b180042db40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:29.143963 (+164867us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:29.168549 (+ 24586us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:29.199858 (+ 31309us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:29.507250 (+307392us) raft_consensus.cc:1671] finished
0426 13:21:29.507275 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:29.532180 (+ 24905us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":8293,"spinlock_wait_cycles":8571136,"thread_start_us":140631,"threads_started":1,"wal-append.queue_time_us":1338855}
I20240426 13:21:29.535996 30894 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.428836 25027 rpcz_store.cc:276] 0426 13:21:28.131309 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:28.146312 (+ 15003us) service_pool.cc:225] Handling call
0426 13:21:28.149066 (+  2754us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:28.149257 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:28.149503 (+   246us) log.cc:842] Serialized 14 byte log entry
0426 13:21:29.142376 (+992873us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:29.198124 (+ 55748us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:29.209487 (+ 11363us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:29.214796 (+  5309us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:29.238994 (+ 24198us) inbound_call.cc:173] Queueing success response
0426 13:21:29.426795 (+187801us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x7b48000b19d8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
Metrics: {"mutex_wait_us":628365,"prepare.queue_time_us":453,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":712986,"raft.run_cpu_time_us":1842,"raft.run_wall_time_us":272240,"spinlock_wait_cycles":109134080,"thread_start_us":348019,"threads_started":2,"wal-append.queue_time_us":296152}
I20240426 13:21:29.614653 30440 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:29.556365 30614 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.049s
I20240426 13:21:29.558182 30718 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
W20240426 13:21:29.612695 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 193824us
I20240426 13:21:29.614903 30631 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:29.593223 30730 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.620491 30730 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.622596 30730 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.608083 30670 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.617197 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:29.624131 25615 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:29.624434 24948 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:29.618405 30974 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:29.625113 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.625138 30974 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:29.625849 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:29.623481 30670 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.626343 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.626101 30730 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:21:29.627207 30979 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20240426 13:21:29.627492 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.627768 30670 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.629570 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.596122 30807 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:29.632593 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 87849.2us
I20240426 13:21:29.634125 30807 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.635664 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:29.636263 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91550.3us
I20240426 13:21:29.637679 30705 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:29.639261 25616 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:29.616750 30893 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:29.640518 25031 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:29.640867 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:29.625949 30974 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:29.608511 30874 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.647485 25621 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:29.646018 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:29.646510 24995 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:29.649020 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.652621 24995 leader_election.cc:401] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:29.652895 25028 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:29.653520 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:29.654795 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.657341 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.611166 30977 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:29.667856 25025 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:29.668974 25096 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:29.669412 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 226946us
I20240426 13:21:29.673429 25026 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20240426 13:21:29.657157 30672 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:29.685372 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.687270 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:29.688372 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52788.4us
I20240426 13:21:29.689172 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:29.689282 30807 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:29.684973 30977 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.689944 30705 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:29.690352 30705 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.690716 30705 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:29.691365 30977 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.691450 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:29.692572 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46259.6us
I20240426 13:21:29.654275 30893 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:29.701030 30893 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:29.703461 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2236) took 1610 ms (client timeout 1643 ms). Trace:
W20240426 13:21:29.703850 25024 rpcz_store.cc:269] 0426 13:21:28.092671 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:28.103846 (+ 11175us) service_pool.cc:225] Handling call
0426 13:21:29.703243 (+1599397us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:29.703395 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2404868864}
I20240426 13:21:29.685189 30672 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.704488 30812 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:29.705680 30876 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.000s
I20240426 13:21:29.667881 30730 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:29.696370 30975 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:29.672247 30815 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:29.710834 25022 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:29.707018 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:29.722543 30894 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.706144 30876 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.725971 30876 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.726969 30665 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.693310 30798 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:29.728549 30602 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.728554 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.728933 30692 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:29.730221 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.730979 30692 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:29.731252 30851 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:29.713773 30815 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.732637 24995 leader_election.cc:398] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:29.732762 30815 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:29.733822 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:29.736976 30778 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:29.738911 30876 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:29.739980 30851 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.740545 30851 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.742419 30851 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.739401 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.742795 30776 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.710227 30672 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.760365 30876 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.760105 30730 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:29.788769 25099 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:29.750718 30604 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:29.748786 30692 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.768116 30876 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.745311 30811 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.797039 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:29.768339 30593 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.789505 30602 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:29.800312 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:29.805191 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.805881 25024 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.802683 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:29.808002 24954 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.789716 30851 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:29.788436 30909 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.810063 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.766391 30815 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:29.812867 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 73523.8us
I20240426 13:21:29.795928 30894 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.818069 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:29.818755 25030 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.758334 30974 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.802543 30672 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:29.828223 24947 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:29.828713 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.799053 30843 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
W20240426 13:21:29.832269 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98571.5us
I20240426 13:21:29.838325 25029 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.839094 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:29.839821 24956 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.848578 25031 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:29.850308 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.851200 30315 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:29.851605 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.853701 30851 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:29.852182 30894 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.863426 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.853900 30614 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:29.861418 30876 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.879321 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.896773 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:29.898407 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:29.909658 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 117907us
W20240426 13:21:29.910602 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59082.6us
I20240426 13:21:29.911162 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.882822 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:29.768344 30730 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.825309 30909 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.878228 30853 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:29.814347 30692 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.896524 30602 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:29.893821 30614 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:29.924938 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.878607 30672 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.927083 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.925832 30853 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:29.912514 30876 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:29.930178 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:29.934072 30721 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:29.939257 30721 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:29.939510 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44016.3us
I20240426 13:21:29.940358 30721 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.941045 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:29.927489 30602 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:29.895718 30851 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.920742 30440 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:29.910293 30705 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.930845 30591 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.935477 30876 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.974094 30876 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:29.939589 30894 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:29.976307 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95028.6us
I20240426 13:21:29.977227 30614 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.977694 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:29.977917 30851 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:29.978480 30851 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:29.980122 30440 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:29.980415 30909 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:29.981113 30440 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:29.981515 30876 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:29.983698 30876 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.984643 30668 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:29.985123 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:29.985257 30705 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:29.993528 24956 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:29.999020 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.937227 30776 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:29.978538 30614 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:29.969483 30847 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:21:29.969358 30602 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.005576 25024 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:30.005677 30404 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:29.932864 30760 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.021730 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.009584 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:30.024685 25025 raft_consensus.cc:2354] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18.
I20240426 13:21:30.010365 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:30.025820 25028 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:30.012025 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:29.933260 30642 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:29.992547 30843 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:30.030109 25070 leader_election.cc:398] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18.
I20240426 13:21:30.030889 25070 leader_election.cc:401] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
W20240426 13:21:30.021561 30556 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.018s	sys 0.008s
W20240426 13:21:30.038302 30556 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.018s	sys 0.009s
I20240426 13:21:30.027252 30593 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:30.047580 30593 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:30.041972 30760 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.048709 30593 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.050515 30760 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.015096 30716 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20240426 13:21:30.052801 30773 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.081s
I20240426 13:21:30.053891 30672 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.054603 30776 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:30.039624 30642 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.023507 30812 raft_consensus.cc:2741] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20240426 13:21:30.021227 30815 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.038790 30807 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:30.061029 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:30.029851 30974 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.030856 30843 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:30.020529 30591 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.077636 30843 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:30.105098 30974 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:30.121809 25030 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:30.059979 30853 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.127617 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.142277 30909 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:30.131022 30692 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.181289 30642 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.187916 25622 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20240426 13:21:30.188552 25622 pending_rounds.cc:76] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 9
I20240426 13:21:30.188868 25622 pending_rounds.cc:97] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20240426 13:21:30.189165 25622 raft_consensus.cc:2866] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:30.174501 30631 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:30.197883 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139108us
I20240426 13:21:30.200103 25028 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:30.204700 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.191560 30909 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:30.208941 30876 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:30.204388 30404 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:30.224562 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 239437us
W20240426 13:21:30.225972 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 301096us
I20240426 13:21:30.226994 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.217654 30876 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:30.233670 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.233661 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.235320 30894 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.235843 30894 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.235992 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:30.237154 30692 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:30.237756 30692 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:30.237740 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:30.238476 30692 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:30.239063 30692 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:30.241566 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:30.242336 25030 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:30.239075 30556 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:30.244673 30556 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:30.245530 30556 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.251075 30692 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:30.253228 30987 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20240426 13:21:30.253058 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:30.253546 30992 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20240426 13:21:30.253813 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:30.251086 25030 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:30.256888 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:30.259254 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 286550us
W20240426 13:21:30.261983 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54321us
I20240426 13:21:30.263332 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.232414 30591 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.225301 30404 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.267102 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.237833 30894 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.254961 30760 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:30.268275 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.268926 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.269644 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.270859 25023 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:30.270609 25028 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:30.307523 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.311578 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.271215 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:30.313795 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.315403 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.314002 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:30.317006 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 21
I20240426 13:21:30.319690 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:30.314520 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:30.323311 30993 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.265s	user 0.003s	sys 0.000s
I20240426 13:21:30.324553 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:30.327121 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.273335 30604 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:30.326488 30894 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:30.331125 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.329330 24956 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:30.333801 30894 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.337284 30894 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.338595 25621 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:30.337181 30798 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:30.340060 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106461us
I20240426 13:21:30.329255 30760 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:30.347916 31000 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:30.347878 30760 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.329371 30672 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20240426 13:21:30.349296 24954 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:30.260527 30909 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:30.378171 31004 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:21:30.378789 30760 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.374081 30665 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:30.342355 30876 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.377599 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:30.367512 31002 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.383870 31002 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.328214 30843 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.320271 30602 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.385829 31002 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:30.404284 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 93025.7us
I20240426 13:21:30.392385 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.410394 24951 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:30.410692 31012 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:30.425505 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:30.388813 31001 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.426077 25100 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:30.391893 30665 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:30.412308 31012 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20240426 13:21:30.430668 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:30.432564 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.440377 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:30.441710 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 208505us
I20240426 13:21:30.444454 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.447710 25104 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:30.442148 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:30.427512 31000 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:30.448596 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:30.449457 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:30.451447 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.454380 24955 raft_consensus.cc:2396] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:21:30.457875 25100 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:30.462025 24951 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:30.462770 25099 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:30.462961 25105 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:30.461503 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.390621 30999 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20240426 13:21:30.464336 25101 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:30.467589 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.464599 25028 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:30.431334 30692 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.465054 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:30.470366 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80482.3us
W20240426 13:21:30.474511 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147442us
I20240426 13:21:30.475107 25027 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:30.464581 30593 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.468940 30834 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:30.468834 30672 raft_consensus.cc:2741] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20240426 13:21:30.492712 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:30.488480 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.483453 25105 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:21:30.489003 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.489583 31010 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:30.489753 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.481190 30404 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.509845 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.510110 25097 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:30.491374 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:30.495136 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44149.6us
I20240426 13:21:30.495855 25100 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:30.497519 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.511684 25099 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:30.512172 25099 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:30.516729 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2945) took 1094 ms. Trace:
I20240426 13:21:30.517258 24950 rpcz_store.cc:276] 0426 13:21:29.414552 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:29.424737 (+ 10185us) service_pool.cc:225] Handling call
0426 13:21:30.508911 (+1084174us) spinlock_profiling.cc:243] Waited 68.7 ms on lock 0x7b540030c0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:30.509002 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158112128}
I20240426 13:21:30.546073 25099 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.433836 31001 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.488664 30834 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:30.552129 25104 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:30.554533 31013 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.046s
W20240426 13:21:30.555950 31005 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20240426 13:21:30.513332 30692 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:30.561838 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.562059 31016 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.562644 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.600510 25100 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:30.562979 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:30.563019 30834 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.601495 25028 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:30.564481 31001 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.566772 30602 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.492285 30593 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.568732 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:30.569448 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 145602us
I20240426 13:21:30.570856 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.557595 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.595654 25097 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:30.605501 25102 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:30.551815 30556 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:30.614151 25105 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:30.604080 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.611568 31013 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:30.618876 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.593963 30870 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:30.600704 30692 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.627094 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.641027 25097 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:21:30.644178 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.645359 30692 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:30.645373 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 157536us
I20240426 13:21:30.613286 30556 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.618527 31013 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:30.619640 30843 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.520655 30665 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.516256 31000 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.678107 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.686671 25623 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
W20240426 13:21:30.687067 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 210439us
I20240426 13:21:30.683477 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.687836 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:30.683979 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46414.1us
I20240426 13:21:30.518412 31010 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.648593 30721 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:30.651815 30556 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:30.693377 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.624290 30760 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20240426 13:21:30.694609 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.696475 25616 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:30.693943 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.684438 30834 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:30.710871 31006 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:30.711099 30977 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:30.711652 30977 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:30.712885 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.713601 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:30.714864 30705 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:30.718103 30556 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.718466 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.720170 30556 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.721401 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.722458 31027 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20240426 13:21:30.739934 31019 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:30.740485 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2260) took 1139 ms. Trace:
I20240426 13:21:30.740862 25029 rpcz_store.cc:276] 0426 13:21:29.601479 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:29.612311 (+ 10832us) service_pool.cc:225] Handling call
0426 13:21:30.740318 (+1128007us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:30.740432 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477828864}
I20240426 13:21:30.740644 31019 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:30.742172 30807 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.676s	user 0.009s	sys 0.000s
I20240426 13:21:30.742560 30807 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18.
I20240426 13:21:30.694169 31015 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:30.748833 31015 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.750545 25098 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:30.641352 30870 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.609354 31016 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.758219 25028 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:30.696548 30798 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20240426 13:21:30.760146 30614 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.739s	user 0.012s	sys 0.000s
I20240426 13:21:30.782563 25022 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:30.796769 24950 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:30.762670 30798 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.796809 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.805704 24954 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:30.809186 25621 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20240426 13:21:30.815418 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:30.822386 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 195372us
W20240426 13:21:30.828348 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 316272us
I20240426 13:21:30.834218 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.838307 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.838973 24954 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:30.805761 30773 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:30.783452 31019 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.839545 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.797830 30614 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:30.797339 31006 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:30.808804 30798 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.840917 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:30.806586 31015 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.844626 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:30.781791 30730 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20240426 13:21:30.850148 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:30.850286 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:30.850980 25104 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:30.846961 25621 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:30.852459 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.853291 24948 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:30.752739 31013 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.840998 30773 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:30.862401 30973 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:30.875269 25621 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:30.864965 31031 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:30.863649 31014 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20240426 13:21:30.875800 25616 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20240426 13:21:30.878943 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:30.880553 30440 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.010s	sys 0.004s
W20240426 13:21:30.882936 30440 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.010s	sys 0.004s
I20240426 13:21:30.875553 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.884182 31033 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:30.855764 30665 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:30.884137 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.897962 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:30.867748 30556 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:30.904973 31016 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.905568 31010 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.906481 30721 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:30.892490 30665 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.907454 30665 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.907716 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:30.908442 25616 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.913273 24948 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:30.918706 30870 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:30.899981 30760 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.020s
I20240426 13:21:30.858863 31006 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.920222 30773 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.921567 30773 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:30.921986 31006 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:30.926470 30894 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.571s	user 0.008s	sys 0.000s
I20240426 13:21:30.852551 30614 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.948452 30894 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.950091 25103 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20240426 13:21:30.928220 30556 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:30.929104 25616 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:30.950793 30556 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.952325 30894 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.939486 30811 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:30.947510 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:30.953429 30556 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.958863 30977 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:30.930796 30692 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:30.988853 30692 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.919833 30760 consensus_queue.cc:1225] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:21:30.966625 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:30.989948 24948 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:30.973742 30811 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:30.992478 30811 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:30.968104 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:30.950284 30614 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:30.981561 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.000280 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.000466 30602 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:21:31.003033 30988 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.862s	user 0.004s	sys 0.002s
I20240426 13:21:30.983078 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.983721 30973 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:30.984381 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:30.985985 30593 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:30.998204 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.012804 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.014678 25029 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:31.016611 25025 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:30.963330 31033 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:31.032430 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.034901 25022 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:31.035154 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.022673 30812 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:31.036602 30812 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:31.017325 30973 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:31.037966 30973 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:31.037846 30812 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.040145 24955 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:31.027850 30977 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:31.040719 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.043092 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.045369 25029 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20240426 13:21:31.050487 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.052523 30974 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:31.046734 30776 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:31.056936 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:31.056877 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.021118 30602 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.026s
I20240426 13:21:31.060634 24955 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
W20240426 13:21:31.061709 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 192962us
I20240426 13:21:31.039623 30614 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.016s
I20240426 13:21:31.058320 30776 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:31.078593 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:31.080067 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.080173 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.080860 25031 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:21:31.049095 30798 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.084221 25026 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:21:31.086333 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:31.086800 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.087088 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.086972 25030 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:31.087569 25589 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:31.087698 25023 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:31.087949 25589 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:31.088886 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.092489 25027 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:31.094149 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:31.096163 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.098091 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:31.105021 31041 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:31.088594 30404 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:31.106241 30668 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:31.106789 30668 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.107868 30798 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:31.108203 30556 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:31.110523 30668 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:31.090096 31029 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:31.093840 30834 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:31.117105 31050 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:31.117710 31050 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.119771 31050 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:31.104758 30776 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.104950 30843 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:31.128448 31033 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.129410 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.130867 24954 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:21:31.133195 24953 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:31.134292 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.137069 25622 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:31.138207 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:31.179656 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:31.186031 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47889.9us
I20240426 13:21:31.176806 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:31.186942 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:31.139667 30834 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:31.139892 30614 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:31.141381 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:31.144158 30995 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.820s	user 0.005s	sys 0.003s
I20240426 13:21:31.149066 25022 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:31.167630 25023 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
W20240426 13:21:31.169083 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69359us
I20240426 13:21:31.170979 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.172914 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:31.175423 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.174834 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:31.179394 31013 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:31.180016 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88711.6us
I20240426 13:21:31.180613 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:31.181552 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:31.186421 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:31.197902 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.195997 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:31.200518 24953 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:31.198489 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:31.202246 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.204484 30692 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:31.196383 31029 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:31.206641 31029 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:31.207054 31029 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:31.199656 31002 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20240426 13:21:31.213892 30973 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.215509 30665 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.191756 30870 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:31.203677 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.217156 30811 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.221097 30556 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:31.221554 30556 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:31.221884 30556 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:31.222402 30811 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.221001 25025 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:31.222950 30692 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:31.224097 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.224251 30692 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.224965 31000 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:31.225534 31000 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:31.226431 31025 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:31.226289 31000 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.224648 30778 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:31.229079 24950 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:31.229228 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.230118 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:31.200819 30853 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:31.232604 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:31.235438 24948 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:31.263896 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:31.203785 30975 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:31.222841 30973 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:31.275017 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.218641 30870 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:31.274645 24995 leader_election.cc:398] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
I20240426 13:21:31.193986 30614 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.233954 30602 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:31.240975 30811 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:31.282939 30665 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.283183 30811 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:31.283592 30811 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:31.285327 30847 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:31.285804 30847 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:31.286592 30847 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.271979 30834 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:31.278672 30870 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:31.288340 30834 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:31.290866 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1199) took 1661 ms. Trace:
I20240426 13:21:31.278313 24995 leader_election.cc:401] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:31.292500 31033 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
I20240426 13:21:31.291323 25615 rpcz_store.cc:276] 0426 13:21:29.629238 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:29.630434 (+  1196us) service_pool.cc:225] Handling call
0426 13:21:29.630762 (+   328us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:29.631099 (+   337us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:29.631127 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:29.669903 (+ 38776us) log.cc:842] Serialized 26 byte log entry
0426 13:21:30.205104 (+535201us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:30.205154 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:30.205243 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:31.143514 (+938271us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:31.290502 (+146988us) raft_consensus.cc:1671] finished
0426 13:21:31.290531 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:31.290780 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6453,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"spinlock_wait_cycles":9583360,"thread_start_us":459292,"threads_started":1,"wal-append.queue_time_us":564917}
I20240426 13:21:31.294008 31022 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20240426 13:21:31.280853 30977 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:31.276425 31002 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:31.300536 31002 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:31.301432 31002 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:31.303809 31008 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.842s	user 0.008s	sys 0.000s
I20240426 13:21:31.304387 25028 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
W20240426 13:21:31.305948 31023 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.620s	user 0.005s	sys 0.000s
I20240426 13:21:31.306066 30870 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.280268 25022 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:21:31.233628 30853 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:31.311233 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.282642 30614 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:31.312379 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:31.219691 31019 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
W20240426 13:21:31.317999 31036 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.495s	user 0.000s	sys 0.004s
I20240426 13:21:31.323063 30874 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:31.318430 30977 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:31.342382 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:31.343672 30977 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:31.345197 30870 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:31.349282 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:31.310230 30974 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.009s
I20240426 13:21:31.355062 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.317992 30811 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.356906 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:31.357595 25617 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:31.358197 30811 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.358563 25621 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:31.358198 24954 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:31.362213 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.362702 24995 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:31.362922 24950 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:31.363147 24995 leader_election.cc:401] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:31.353426 30977 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:31.411976 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:31.346967 30665 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:31.428304 30665 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.418973 30556 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.424180 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.431710 24952 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:31.453579 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.413666 30602 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:31.430476 30642 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:31.465736 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:31.466810 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 237626us
I20240426 13:21:31.480217 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.481006 24953 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:31.481436 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:31.457161 30977 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20240426 13:21:31.502425 25027 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20240426 13:21:31.510366 24950 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:31.511049 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.515413 31033 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:31.518770 30977 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:31.519213 30977 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:31.520208 30692 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:31.521689 30977 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.523169 30977 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:31.524605 31029 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:31.524806 30812 consensus_queue.cc:1159] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 18 status { last_received { term: 16 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:31.525907 30834 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.526582 31029 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:31.526904 30834 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:31.535337 31061 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.065s
I20240426 13:21:31.484862 30642 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.536149 31061 consensus_queue.cc:1225] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 16.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20240426 13:21:31.537513 30642 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.539868 25618 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:31.555279 24955 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20240426 13:21:31.551712 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.556317 24953 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:31.563520 25023 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:31.564919 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.565729 25031 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:31.574292 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:31.541450 30315 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20240426 13:21:31.587960 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:31.594003 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:31.587538 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.676754 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:31.586427 30977 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
W20240426 13:21:31.709625 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 122706us
I20240426 13:21:31.586503 31021 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:31.678388 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 91482us
I20240426 13:21:31.719058 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:31.719192 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131745us
I20240426 13:21:31.724103 31050 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:31.724542 31050 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:31.724907 31050 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:31.727162 30716 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:31.727617 30716 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.721541 30977 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:31.728551 31021 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.636770 30730 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20240426 13:21:31.730062 25024 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:31.734354 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
W20240426 13:21:31.723881 31066 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20240426 13:21:31.588531 31033 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.131s
I20240426 13:21:31.743101 25620 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20240426 13:21:31.743815 31067 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:31.744215 24951 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20240426 13:21:31.745949 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:31.740149 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:31.718791 30556 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.722064 30665 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.753509 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:31.729549 30977 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.740737 31006 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:31.749120 31013 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:31.788900 31068 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:31.791415 31071 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20240426 13:21:31.797164 31054 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.501s	user 0.000s	sys 0.003s
I20240426 13:21:31.806946 31021 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.757920 31063 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:31.778059 31033 consensus_queue.cc:1225] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:21:31.822537 31063 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:31.813954 31013 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.849939 31013 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:31.852475 31063 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:31.853029 30602 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:21:31.860780 30716 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:31.848248 31050 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.866047 31050 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:31.866909 31069 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20240426 13:21:31.878823 30760 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:31.879257 30760 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:31.880148 30760 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:31.881347 30760 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:32.082165 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 351348us
I20240426 13:21:32.084899 30614 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.085480 30614 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.085407 31078 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.087461 31078 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.087818 30614 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.090124 25023 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:32.091290 31078 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.106482 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 811) took 1055 ms. Trace:
I20240426 13:21:32.106909 25029 rpcz_store.cc:276] 0426 13:21:31.051642 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:31.052080 (+   438us) service_pool.cc:225] Handling call
0426 13:21:31.062043 (+  9963us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:31.076668 (+ 14625us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:31.077327 (+   659us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:31.296701 (+219374us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:31.297146 (+   445us) log.cc:842] Serialized 26 byte log entry
0426 13:21:31.503793 (+206647us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:31.519072 (+ 15279us) log.cc:842] Serialized 14 byte log entry
0426 13:21:31.519302 (+   230us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:31.519374 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:32.106205 (+586831us) raft_consensus.cc:1671] finished
0426 13:21:32.106301 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:32.106430 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15755,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":29,"raft.queue_time_us":1227,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":12570,"spinlock_wait_cycles":11898752,"thread_start_us":206292,"threads_started":1,"wal-append.queue_time_us":425254}
I20240426 13:21:32.125584 30812 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:32.125977 30812 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:32.126659 30812 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.127391 30812 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:32.141851 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2266) took 1878 ms (client timeout 2076 ms). Trace:
W20240426 13:21:32.142393 25024 rpcz_store.cc:269] 0426 13:21:30.262961 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:30.263080 (+   119us) service_pool.cc:225] Handling call
0426 13:21:32.141663 (+1878583us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000b0754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:32.141791 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3371629952}
I20240426 13:21:32.157177 25099 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.161476 25099 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
W20240426 13:21:32.161863 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2252) took 1691 ms (client timeout 1907 ms). Trace:
W20240426 13:21:32.162191 25099 rpcz_store.cc:269] 0426 13:21:30.470976 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:30.475489 (+  4513us) service_pool.cc:225] Handling call
0426 13:21:31.300771 (+825282us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b18000f8c40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90352b46 00007fda902a50fe 00007fda9030e4f7 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:21:31.423869 (+123098us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b580051b6b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:21:32.157049 (+733180us) spinlock_profiling.cc:243] Waited 7.3 ms on lock 0x7b580051b3b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:21:32.161781 (+  4732us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1282960512}
W20240426 13:21:32.165593 31055 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.883s	user 0.003s	sys 0.001s
I20240426 13:21:32.167343 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2419) took 1035 ms. Trace:
I20240426 13:21:32.167712 24947 rpcz_store.cc:276] 0426 13:21:31.132327 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:31.132455 (+   128us) service_pool.cc:225] Handling call
0426 13:21:31.132811 (+   356us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:31.133218 (+   407us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:31.133270 (+    52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:31.133793 (+   523us) log.cc:842] Serialized 26 byte log entry
0426 13:21:31.201037 (+ 67244us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:31.229233 (+ 28196us) log.cc:842] Serialized 14 byte log entry
0426 13:21:31.229450 (+   217us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:31.229539 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:32.165175 (+935636us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:32.167062 (+  1887us) raft_consensus.cc:1671] finished
0426 13:21:32.167170 (+   108us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:32.167290 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18211,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":387,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":33300,"spinlock_wait_cycles":61333760,"thread_start_us":63012,"threads_started":1,"wal-append.queue_time_us":138823}
I20240426 13:21:32.189971 31022 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:32.202353 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2412) took 1137 ms. Trace:
I20240426 13:21:32.202808 24955 rpcz_store.cc:276] 0426 13:21:31.066077 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:31.078223 (+ 12146us) service_pool.cc:225] Handling call
0426 13:21:32.202170 (+1123947us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:32.202297 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069189504}
I20240426 13:21:32.217085 30853 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.222550 30812 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.223112 30812 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.225134 30812 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.233105 31078 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:32.235435 31078 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:32.239933 31078 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:32.241883 30977 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.244544 30977 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:32.246519 30977 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:32.242295 30812 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.247839 30812 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.252535 31078 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:32.257334 31078 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.263549 30848 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.297559 30848 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.299629 30848 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.284319 30843 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:32.286466 25028 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:21:32.304390 30843 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.259501 30812 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.313864 30847 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20240426 13:21:32.321612 31078 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:32.327451 30843 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.329191 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.331406 30773 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.331355 30812 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.331986 30773 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.332136 30812 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.333776 30773 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.334254 30812 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.341504 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.342180 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:32.343261 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.344002 30614 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:32.344403 30614 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:32.344699 30614 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:32.359783 30614 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.373939 30614 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.401726 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:32.408757 25615 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:32.409785 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:32.412261 30811 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240426 13:21:32.417371 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.60907e+06us
I20240426 13:21:32.425180 25027 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20240426 13:21:32.426936 30853 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:32.433387 31000 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.433969 31000 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.435920 31000 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.436173 31006 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.436674 31006 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.437669 31000 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.438195 31000 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.438763 31006 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:32.440209 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 694269us
I20240426 13:21:32.440652 30778 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:32.441035 30778 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.441375 30778 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.443452 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.444133 25620 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:32.445818 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.447526 25031 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:21:32.448117 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.448740 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.449486 25029 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.452870 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.449415 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.453531 25028 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:32.453743 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:32.454903 25022 raft_consensus.cc:2396] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20240426 13:21:32.455366 31000 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.457886 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.458652 24955 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:32.456709 24951 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20240426 13:21:32.459861 24951 pending_rounds.cc:76] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:21:32.460209 24951 pending_rounds.cc:97] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20240426 13:21:32.460506 24951 raft_consensus.cc:2866] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:32.462553 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:32.463142 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.468369 30440 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.468447 24951 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:32.469004 30440 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.471381 30440 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.474005 24955 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20240426 13:21:32.474999 31002 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.475142 30837 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.475574 31002 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.477062 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:32.477754 31002 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.475642 30837 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.481142 30837 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.482061 30851 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.482648 30851 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.483002 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.484192 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:32.484838 30851 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.485404 30851 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.485806 30851 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:21:32.486567 24854 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 178) took 20180 ms (client timeout 15000 ms). Trace:
I20240426 13:21:32.486661 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.487608 31013 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.488210 30848 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:32.488348 31013 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.488960 30848 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:32.488561 24853 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.490101 30848 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.492013 30556 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:32.487676 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.493135 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.490162 24853 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 9 to 15. New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.494920 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.492511 30556 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.497891 30556 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:32.498987 30730 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:32.495724 24955 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:32.501060 31050 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:32.501541 31050 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.501994 31050 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:32.507730 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.520437 24955 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:32.520624 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.521579 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.521755 25027 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:32.513104 25029 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:32.522469 24947 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:21:32.531867 25030 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20240426 13:21:32.520635 24853 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 9 to 14, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 14 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:32.535400 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.536463 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:32.537081 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.539237 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.539526 24853 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.537854 25025 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:32.540076 24951 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:32.541435 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.490306 31013 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.545513 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.507216 30975 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:21:32.551394 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.552558 25025 raft_consensus.cc:2396] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20240426 13:21:32.548741 24853 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 9, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 9 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:32.532712 30837 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:32.555415 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 136963us
I20240426 13:21:32.549607 24947 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Deduplicated request from leader. Original: 14.13->[15.14-15.14]   Dedup: 15.14->[]
I20240426 13:21:32.547705 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.556286 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 11.
I20240426 13:21:32.542187 24955 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:32.546341 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:32.557837 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.558898 24947 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20240426 13:21:32.559541 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.560613 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 11, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:21:32.486977 24854 rpcz_store.cc:269] 0426 13:21:12.306408 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:12.310878 (+  4470us) service_pool.cc:225] Handling call
0426 13:21:32.486485 (+20175607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:29.599892 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:29.615306 (+ 15414us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:29.615359 (+    53us) write_op.cc:427] Acquired schema lock
0426 13:21:29.615383 (+    24us) tablet.cc:660] Decoding operations
0426 13:21:29.619078 (+  3695us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:29.619177 (+    99us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:29.619210 (+    33us) tablet.cc:683] Acquiring locks for 20 operations
0426 13:21:29.620806 (+  1596us) tablet.cc:699] Row locks acquired
0426 13:21:29.620842 (+    36us) write_op.cc:260] PREPARE: finished
0426 13:21:29.620983 (+   141us) write_op.cc:270] Start()
0426 13:21:29.621128 (+   145us) write_op.cc:275] Timestamp: P: 1714137689620925 usec, L: 0
0426 13:21:29.621157 (+    29us) op_driver.cc:331] REPLICATION: starting
0426 13:21:29.621814 (+   657us) log.cc:842] Serialized 7694 byte log entry
0426 13:21:30.640665 (+1018851us) op_driver.cc:444] REPLICATION: finished
0426 13:21:31.020716 (+380051us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8d3cee2e 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda93d28980
0426 13:21:31.105095 (+ 84379us) write_op.cc:300] APPLY: starting
0426 13:21:31.105202 (+   107us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:31.424094 (+318892us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:31.424126 (+    32us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:31.833010 (+408884us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:31.834449 (+  1439us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=40,mrs_lookups=0
0426 13:21:31.834504 (+    55us) write_op.cc:311] APPLY: finished
0426 13:21:31.834920 (+   416us) log.cc:842] Serialized 180 byte log entry
0426 13:21:31.841835 (+  6915us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:31.842261 (+   426us) write_op.cc:446] Released schema lock
0426 13:21:31.842354 (+    93us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:31.842628 (+   274us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:31.842649 (+    21us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1055958,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":412341,"apply.run_cpu_time_us":31995,"apply.run_wall_time_us":740145,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87767,"num_ops":20,"prepare.queue_time_us":296615,"prepare.run_cpu_time_us":19528,"prepare.run_wall_time_us":216844,"raft.queue_time_us":221323,"raft.run_cpu_time_us":31394,"raft.run_wall_time_us":490952,"replication_time_us":1002211,"spinlock_wait_cycles":52475776,"thread_start_us":603625,"threads_started":5,"wal-append.queue_time_us":379925,"wal-append.run_cpu_time_us":21079,"wal-append.run_wall_time_us":1472662}]]}
I20240426 13:21:32.535686 30440 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20240426 13:21:32.565078 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.561753 30848 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:32.566663 30848 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:32.567030 30848 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.570263 30556 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.572294 31002 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:21:32.558003 30837 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.574904 24853 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 11 to 13, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 13 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.557559 30975 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:32.582819 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.584749 24948 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:21:32.586436 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.588757 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.589880 24853 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.592331 25615 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20240426 13:21:32.592687 25615 pending_rounds.cc:76] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 13
I20240426 13:21:32.592821 25615 pending_rounds.cc:97] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20240426 13:21:32.592973 25615 raft_consensus.cc:2866] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:32.564579 30665 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:32.564117 30851 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.589540 31013 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:32.588582 30837 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:32.600900 31013 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:32.601074 23801 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd060 after lost signal to thread 25069
I20240426 13:21:32.621834 24853 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.601614 31033 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:32.630698 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 11 to 13, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 13 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:21:32.621668 23801 kernel_stack_watchdog.cc:198] Thread 25069 stuck at /home/jenkins-slave/workspace/build_and_test@3/src/kudu/rpc/outbound_call.cc:308 for 827ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20240426 13:21:32.619427 31013 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.588380 30848 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.638033 30848 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.644377 30556 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.644460 31013 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.647265 31013 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.647768 25619 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20240426 13:21:32.651527 31050 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.657964 30843 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.648492 30975 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.658614 30843 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.633827 31033 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.661213 31033 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:32.652647 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51218.6us
I20240426 13:21:32.653123 30874 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:32.664381 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.665216 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.665745 25022 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:32.666287 30843 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.667285 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.601714 30778 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.668118 30843 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:32.668510 30843 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:32.668850 30843 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.668968 31034 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:32.669402 31034 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:32.670280 31034 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.672811 31009 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.673374 31009 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.674306 30707 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:32.674749 30707 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:32.673689 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 117688us
I20240426 13:21:32.680282 30893 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.680790 31016 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:32.681231 31016 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.680794 30893 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.681614 31016 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:32.683413 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.688927 25031 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:32.690508 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.691274 31001 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.691766 31001 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.693746 30798 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.694286 30798 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.696178 30705 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.688967 31050 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.697124 30631 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.697789 30631 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.699611 30670 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.700242 30670 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.702492 30893 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.702997 30893 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.704208 31090 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.704574 30707 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.705075 30707 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.704830 31090 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.705375 31009 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.705464 30707 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.706573 31009 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.706712 31090 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.707289 31090 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.708202 30807 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:32.708600 30811 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:32.708695 30807 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.710255 30614 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:32.710609 30807 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.704268 30778 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.703606 30837 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.712280 25098 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:32.712692 30876 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.713151 30876 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.713598 30837 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:32.702255 30670 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.715067 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:32.716122 31089 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:32.716351 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.716898 25098 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:32.716637 31089 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.718462 31089 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.710630 30614 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:32.719928 30614 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.721163 30614 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:32.721199 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56889.5us
I20240426 13:21:32.721705 30665 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:32.722365 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.722666 30665 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.723677 30874 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:32.724089 30874 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:32.724378 30668 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:32.724977 30668 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.696741 30705 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.725375 30668 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:32.726739 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.727743 31003 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:32.703333 30593 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:32.729825 30593 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.651212 24853 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 10 to 13, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 13 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:32.682898 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:32.704162 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.98263e+06us
I20240426 13:21:32.734171 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:32.724427 30874 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.735919 25105 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:32.737505 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.738157 25099 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:32.742094 25027 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:32.742704 25025 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:32.711763 30807 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.707017 31009 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.749574 25098 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:32.753952 25028 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:32.758975 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.759703 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:32.758996 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.761232 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:32.764125 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.707720 31090 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.736750 30705 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.765942 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.764791 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:32.769690 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.770848 24853 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 10. New cstate: current_term: 10 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.771857 25098 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:21:32.752774 30853 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:32.752192 31034 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.743739 31027 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:32.772990 25105 pending_rounds.cc:76] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 9
I20240426 13:21:32.798105 25105 pending_rounds.cc:97] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20240426 13:21:32.781893 25615 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 17.14->[18.15-18.15]   Dedup: 18.15->[]
I20240426 13:21:32.798683 25105 raft_consensus.cc:2866] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:32.761832 30977 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:32.769593 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.777463 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.703408 30893 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.767398 30909 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:32.800078 25102 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:32.802337 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.802969 25098 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:32.803550 25101 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.760888 30807 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.771031 30811 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:32.776613 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.799016 25615 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20240426 13:21:32.767470 31089 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:32.806866 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.769768 30631 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:32.812382 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 247348us
I20240426 13:21:32.774948 30798 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.812877 25103 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:32.817045 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.811514 30974 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:32.780342 30593 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.816102 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.819959 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.814123 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.821695 25097 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:32.815656 24853 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 11. New cstate: current_term: 11 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:32.800896 30977 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.769881 30896 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20240426 13:21:32.804694 30707 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.805346 30876 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:32.826069 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:32.819685 30974 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.826548 30778 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:32.832049 30893 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.810485 30909 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.834224 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.822989 24853 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:32.856114 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.841548 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:32.845125 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:32.817049 31089 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.852802 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.853767 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.811748 30811 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.860219 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.801210 30853 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:32.856762 25098 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:32.849838 24954 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:32.858129 24950 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:32.819919 30602 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:32.866526 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.827015 30977 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:32.865950 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.869764 24950 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:32.855765 30974 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.870055 24956 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:32.869941 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.877388 24951 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:32.877549 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:32.876943 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 144240us
I20240426 13:21:32.868444 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.862967 30909 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:32.882609 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.883172 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.855909 30778 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.883859 25024 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:32.861047 31089 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:32.885298 25103 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:21:32.885875 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:32.860958 30440 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:32.869665 30811 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:32.865782 31033 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:32.901985 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.901934 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.871444 30602 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:32.910969 25024 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:32.913509 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.913251 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.918252 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.921643 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.923806 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.924505 24951 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:32.925360 25102 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:32.927138 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.927498 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.927138 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.933122 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:32.887426 30778 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:32.937851 25029 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:32.938103 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.939081 25098 raft_consensus.cc:2396] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20240426 13:21:32.939518 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.938068 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 8 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 11 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:32.942934 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:32.905313 30440 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:32.945078 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:32.976156 30440 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:32.916415 30602 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:32.981118 25621 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:32.981485 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:32.981719 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:32.983059 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 387503us
I20240426 13:21:32.987761 30807 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:32.988291 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.988741 24853 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 6 to 10, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 10 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:32.990234 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.992158 31027 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:32.992848 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:32.996474 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080021cbc0 after lost signal to thread 23796
W20240426 13:21:33.033890 25126 debug-util.cc:398] Leaking SignalData structure 0x7b0800194360 after lost signal to thread 23799
W20240426 13:21:32.997177 31103 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20240426 13:21:32.997663 31090 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.049213 30853 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.049480 24853 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 8 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 11 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:33.050850 24853 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:33.062781 25101 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:33.075440 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.006618 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.016105 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.035104 25098 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:33.035552 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:33.035111 25126 debug-util.cc:398] Leaking SignalData structure 0x7b0800088060 after lost signal to thread 24888
I20240426 13:21:33.041672 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.044270 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:32.978555 30668 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.073218 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.006315 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:33.084051 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08002d50a0 after lost signal to thread 24980
I20240426 13:21:33.084364 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:21:33.085284 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08001127a0 after lost signal to thread 25055
W20240426 13:21:33.085755 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080014dc00 after lost signal to thread 25069
W20240426 13:21:33.087112 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08003b32c0 after lost signal to thread 25129
I20240426 13:21:33.089360 25103 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:33.075446 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:33.090297 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:33.097513 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080015be20 after lost signal to thread 25650
W20240426 13:21:33.037189 31087 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20240426 13:21:33.106827 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 90813.4us
I20240426 13:21:33.052852 30843 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:33.108258 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49997.6us
I20240426 13:21:33.116832 24948 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
W20240426 13:21:33.117873 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 246778us
I20240426 13:21:33.037442 31009 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.143011 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.088069 31027 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.145073 31108 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.145648 31108 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.146682 31027 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:33.145365 31024 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:33.147264 31027 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:33.147580 31024 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:33.149395 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.150861 24954 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20240426 13:21:33.151427 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.123208 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.105785 30876 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.126303 25097 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 19. Ops: []
I20240426 13:21:33.124573 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:33.125129 31101 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.130246 25096 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:33.160249 25102 raft_consensus.cc:595] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.163233 25102 raft_consensus.cc:603] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:33.161751 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.161255 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.165613 24955 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:33.120112 31006 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:33.164774 24947 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:33.174214 24954 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:33.139480 30811 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.108783 31001 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:33.184208 24948 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:33.148010 31027 consensus_queue.cc:1225] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 22.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20240426 13:21:33.153393 30614 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.176123 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.186163 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.110308 30874 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.193930 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.206902 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.176992 31006 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.214475 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.209074 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:33.211341 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:33.218953 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 341989us
I20240426 13:21:33.219137 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.219696 25097 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:33.188314 31108 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:33.231209 25031 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:33.175674 31104 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20240426 13:21:33.230762 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.244810 25621 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:33.175007 30837 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20240426 13:21:33.174777 30876 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.171872 31101 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.175720 31089 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.266878 31089 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.212059 31001 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:33.252298 31102 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:33.254870 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.255318 30896 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:33.217650 31006 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:33.254021 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.257663 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.217916 31027 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20240426 13:21:33.268790 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:33.273633 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:33.247066 31033 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.192965 30614 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.209908 30893 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20240426 13:21:33.255795 30876 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:33.252601 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 323230us
I20240426 13:21:33.390189 30876 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:33.390621 30876 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.391522 30876 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.393575 30876 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:33.267297 30843 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.400810 30843 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:33.320823 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.325423 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:33.413936 25097 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.414814 25097 raft_consensus.cc:603] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:33.336151 31090 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:33.256402 31108 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20240426 13:21:33.279474 31089 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20240426 13:21:33.341876 30602 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.275658 30672 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.343598 30811 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.345273 30874 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.275314 31001 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:33.346060 30668 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.346635 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:33.348526 31034 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.003s
I20240426 13:21:33.275187 31102 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.349694 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.349982 31101 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.354043 30974 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.433346 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:33.355660 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46864.4us
I20240426 13:21:33.355834 30778 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.272714 30896 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.358366 30591 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:33.358418 30909 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.360460 30614 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:33.362445 30631 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:33.457648 30631 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.365378 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.458389 30631 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.459264 31102 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.459425 30631 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:33.376204 31010 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:33.460280 31108 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:33.460633 31010 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.461222 31010 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:33.381912 25026 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:33.420456 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.464972 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.465098 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.465584 25026 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:33.466257 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.466429 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.467378 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.467548 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.383411 31118 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.324440 25024 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:33.432395 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.468446 31129 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.469262 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.445695 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:33.447788 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 257499us
I20240426 13:21:33.449368 25098 raft_consensus.cc:2396] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
W20240426 13:21:33.450695 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 149735us
I20240426 13:21:33.469931 25024 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.470431 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:33.444386 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:33.452911 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.471810 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:33.421437 31116 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.080s	user 0.000s	sys 0.004s
W20240426 13:21:33.421648 31105 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.146s	user 0.000s	sys 0.004s
W20240426 13:21:33.439301 31112 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.174s	user 0.007s	sys 0.002s
W20240426 13:21:33.425415 31092 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s	user 0.002s	sys 0.016s
I20240426 13:21:33.448429 30896 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:33.458194 31135 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.458941 25028 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.367086 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.460661 31108 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.461261 30893 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:33.434114 30672 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.469677 25027 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:33.437878 25104 raft_consensus.cc:2362] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18. Re-sending same reply.
I20240426 13:21:33.468508 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.452205 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.490072 25025 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.473934 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:33.442564 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 266602us
I20240426 13:21:33.448678 30591 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:33.448783 31109 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.218s	user 0.000s	sys 0.005s
W20240426 13:21:33.450028 31113 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.097s	user 0.007s	sys 0.000s
W20240426 13:21:33.449898 31114 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.098s	user 0.002s	sys 0.005s
I20240426 13:21:33.477087 25100 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20240426 13:21:33.477885 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.478257 25026 raft_consensus.cc:2354] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:21:33.479247 25096 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:33.480855 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:33.476619 31016 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.795s	user 0.010s	sys 0.001s
W20240426 13:21:33.524945 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47129.7us
I20240426 13:21:33.485423 31001 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:33.486456 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:33.486994 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 302376us
I20240426 13:21:33.526934 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.469050 31129 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.478379 31102 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:33.494318 25621 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:21:33.530371 25619 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [16.15-16.15]
W20240426 13:21:33.531054 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50263.6us
I20240426 13:21:33.531657 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.493683 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:33.495607 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.519153 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.532743 24921 leader_election.cc:401] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:33.532982 25620 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:33.533524 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.520382 30778 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.518318 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:33.523141 25100 pending_rounds.cc:76] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 14
I20240426 13:21:33.555013 25100 pending_rounds.cc:97] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20240426 13:21:33.555423 25100 raft_consensus.cc:2866] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:33.534807 30602 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.535180 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.563088 30975 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:33.563623 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.563900 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.564321 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.564599 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.565032 24955 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:33.565758 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.566215 24955 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:33.536443 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.536976 31024 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:33.537420 31102 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:33.569000 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.569217 31102 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.555145 30593 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:33.538376 31129 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.572655 30909 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.538944 31013 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.573235 30672 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.573345 31013 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.575237 31013 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.468375 31118 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.576287 31013 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:33.577066 31118 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.577667 31118 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:33.579128 25023 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:33.532217 31016 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.580176 31134 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:33.581050 30778 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:33.581418 30778 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:33.581804 31016 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.582053 30778 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.539260 25102 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:33.539317 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.583966 24948 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:33.584435 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.481629 31089 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20240426 13:21:33.539530 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.539752 30707 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:33.585690 24947 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:33.540971 30974 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.585775 30778 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:33.540093 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.586622 24951 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:33.542332 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.487591 30893 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.584709 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.543123 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:33.543233 30811 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:33.544207 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.544595 31006 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.492172 31121 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:33.545336 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.546002 31108 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.521289 31120 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.026s
I20240426 13:21:33.610251 31108 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:33.548465 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.529340 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:33.612363 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 83079.4us
I20240426 13:21:33.613960 30404 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:33.550021 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.550292 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.615298 24954 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:33.529660 31003 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.616398 31003 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.550944 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.554003 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.550287 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.561265 31027 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240426 13:21:33.523093 30977 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.641s	user 0.009s	sys 0.000s
I20240426 13:21:33.561877 31078 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:33.621145 30977 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:21:33.621639 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67719.3us
I20240426 13:21:33.621827 31078 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:33.622251 31078 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:33.623366 25022 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.623067 31078 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.624236 31078 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:33.624926 31027 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:33.532464 30440 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.625370 25024 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:33.626705 30440 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:33.562356 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.582554 25105 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:33.585031 25102 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:33.629002 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.629129 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.629618 25102 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:33.630379 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.631335 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.621285 30798 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:33.632611 30798 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.626717 24950 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12. Ops: []
W20240426 13:21:33.634655 23801 debug-util.cc:398] Leaking SignalData structure 0x7b0800355c80 after lost signal to thread 25069
I20240426 13:21:33.634238 30896 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.588091 25103 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:33.636384 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:33.636691 25096 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:33.589152 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.637185 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:21:33.637670 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48589us
I20240426 13:21:33.638027 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.638332 30811 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:33.638439 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.638748 30811 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.638916 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:33.639114 30811 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:33.628219 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.541690 30668 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:33.623492 25070 leader_election.cc:398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:21:33.623090 30977 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:33.591215 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:33.640978 31027 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:33.641379 31027 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.638850 30974 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:33.641676 31027 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:33.641909 30974 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.642235 30974 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:33.642513 31016 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:33.643584 31016 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:33.589874 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.644439 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.605845 30893 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.639699 31090 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:33.602751 30672 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:33.645860 30672 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.646198 30672 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:33.646333 30593 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:33.646764 30593 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.647487 30593 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.607523 24953 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:33.637213 31010 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.648622 30593 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:33.649137 25097 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:33.609093 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.615371 31089 consensus_queue.cc:1225] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20240426 13:21:33.650578 25099 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:33.650770 25097 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:33.651629 25099 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:33.561374 30874 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:33.618310 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.652747 31024 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:33.645553 30893 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:33.533169 31001 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.620221 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:33.620589 31003 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.633404 30798 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.639835 25100 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:33.640431 25070 leader_election.cc:401] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:21:33.591718 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48665.3us
I20240426 13:21:33.645536 31090 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.606623 31006 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:33.654266 25618 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:33.651949 25099 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:33.682570 25099 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.684310 25099 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:33.655611 25105 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:33.655970 24996 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:33.685403 24996 leader_election.cc:401] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:33.657196 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.686578 25096 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.686698 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.687276 25096 raft_consensus.cc:603] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:21:33.588213 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 94714us
I20240426 13:21:33.600880 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:33.664018 25104 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
W20240426 13:21:33.677968 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54610.7us
I20240426 13:21:33.689929 31090 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.652701 30874 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:33.695639 25025 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20240426 13:21:33.679999 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.697345 25022 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:33.653406 31010 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.696727 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
I20240426 13:21:33.707053 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.708184 25023 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.704499 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.708658 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:33.703847 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:33.661540 31001 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:33.488782 31135 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.677681 30909 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:33.717743 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61890us
I20240426 13:21:33.717818 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.717118 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:33.728694 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 114352us
I20240426 13:21:33.729215 25099 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:33.730573 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.720158 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:33.697809 30876 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:33.695971 30874 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:33.722365 30614 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 18
I20240426 13:21:33.699714 31090 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:33.692283 30631 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:21:33.710983 30798 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:33.663403 30591 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.735733 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.740415 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.740767 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.750372 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.750957 25104 raft_consensus.cc:2362] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20. Re-sending same reply.
I20240426 13:21:33.751672 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.718410 31126 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:33.731243 25099 raft_consensus.cc:2362] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17. Re-sending same reply.
I20240426 13:21:33.720953 30843 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:21:33.737188 30707 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20240426 13:21:33.726810 31040 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:33.758535 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.727756 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:33.760294 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.719641 31024 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20240426 13:21:33.762617 25104 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:33.745970 30977 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:33.746219 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:33.733304 30315 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20240426 13:21:33.765774 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.766556 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.768016 25104 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:33.770570 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.771064 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:21:33.776130 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.774017 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.777318 25030 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:33.776239 25096 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:33.754473 31126 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.747206 30668 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
I20240426 13:21:33.758908 30591 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:33.778446 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:33.793066 25024 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:33.788851 30896 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:33.779752 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:33.745890 31131 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20240426 13:21:33.779872 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:33.767611 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.609s (SENT) [suppressed 10 similar messages]
W20240426 13:21:33.812562 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44982us
I20240426 13:21:33.786576 31034 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Stepping down as leader of term 11
I20240426 13:21:33.787241 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.813674 31034 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:33.763548 31010 consensus_queue.cc:1159] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 22 status { last_received { term: 22 index: 15 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: [21.15-21.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:33.771107 30975 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:33.814967 30975 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:33.815297 30975 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:33.815951 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 129360us
I20240426 13:21:33.815877 30665 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:33.816730 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.817404 31145 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:33.771982 31124 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:33.774493 31013 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:33.780097 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:33.814513 31034 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.745587 31115 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20240426 13:21:33.817996 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.779877 30977 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:33.818214 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:33.842751 25102 raft_consensus.cc:2396] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20240426 13:21:33.818810 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:33.791179 30602 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:33.791446 31126 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:33.841104 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:33.848409 25097 raft_consensus.cc:595] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.848855 25097 raft_consensus.cc:603] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:33.844203 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.852671 31115 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:33.853201 31115 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:33.854012 31115 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.841753 31134 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20240426 13:21:33.842947 30315 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20240426 13:21:33.772006 31117 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
W20240426 13:21:33.862752 31138 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20240426 13:21:33.863207 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:33.865777 31140 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20240426 13:21:33.845530 25102 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:21:33.866875 31135 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:33.868449 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 119493us
W20240426 13:21:33.870414 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90444.1us
I20240426 13:21:33.870443 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.871923 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.873019 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.819132 31089 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:33.875746 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:33.877322 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.877846 30975 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.880657 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.883397 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.884413 24955 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:33.894418 24955 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:33.894590 24950 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:33.896342 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.897109 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.898798 24947 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:33.899909 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:33.900342 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.901840 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.903523 24954 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:33.903720 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.908777 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.908906 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.874951 30593 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:33.913389 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.911475 24956 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:33.874524 31145 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20240426 13:21:33.916632 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 87122.9us
I20240426 13:21:33.922061 31145 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:33.924867 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.880236 31016 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:33.879673 30977 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:33.889336 30665 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:33.903131 31108 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:33.879244 31135 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:33.927198 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.882931 30876 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:33.933935 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:33.879949 31024 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:33.878697 30631 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:33.941579 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dacba3e2733347ecb1304d307310c944"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:33.847896 30672 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:33.933537 31034 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:33.880017 31013 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20240426 13:21:33.900552 31124 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:33.943089 25618 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:33.867247 31027 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:33.943120 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:33.990517 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:33.927836 31006 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:33.970451 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72328.3us
I20240426 13:21:33.987283 31133 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:33.994203 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:33.994786 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 109335us
I20240426 13:21:33.938145 30665 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:33.999053 24955 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:33.962697 31108 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:33.916232 30837 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.056s
I20240426 13:21:34.001219 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:34.003330 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89881.6us
I20240426 13:21:34.006866 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:34.010257 25031 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:33.992480 31103 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20240426 13:21:33.989171 31127 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:21:34.030333 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.032298 25103 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:34.034739 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.047211 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:34.048475 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.049791 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.051499 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.052994 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.054327 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:33.998296 31024 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:34.063584 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:34.140859 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106440us
I20240426 13:21:33.940838 30974 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.010927 31108 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:34.001639 30665 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:34.029724 30631 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:21:34.170344 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 175080us
I20240426 13:21:34.011566 30837 consensus_queue.cc:1225] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:21:34.036644 31016 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:34.041543 31133 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:34.048641 30876 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20240426 13:21:34.183492 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 120002us
I20240426 13:21:34.183581 31118 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:34.142570 30593 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:34.173539 31006 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20240426 13:21:34.185895 31116 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20240426 13:21:34.178046 30315 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:34.176899 30909 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20240426 13:21:34.182459 30893 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s	user 0.008s	sys 0.001s
I20240426 13:21:34.178562 31135 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:34.175388 31167 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:34.181284 30760 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:34.197124 30631 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:34.196345 31118 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:34.165868 31024 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.198637 30614 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
W20240426 13:21:34.234184 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 178582us
I20240426 13:21:34.174041 30977 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.206877 30975 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:34.207147 31016 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:34.234671 31013 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:34.258693 31167 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:34.269134 31024 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:34.269515 30778 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.001s
I20240426 13:21:34.229044 30440 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:34.274718 30672 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:34.271154 30798 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.523s	user 0.006s	sys 0.000s
I20240426 13:21:34.267546 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:34.279053 31126 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.196607 31124 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.236896 30760 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.283387 25098 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20240426 13:21:34.267275 31078 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.001s
I20240426 13:21:34.330948 24949 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20240426 13:21:34.334389 25100 raft_consensus.cc:595] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.341771 25100 raft_consensus.cc:603] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:21:34.267086 31001 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.551s	user 0.004s	sys 0.004s
W20240426 13:21:34.335098 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 67619.5us
I20240426 13:21:34.343183 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.284152 31133 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:34.284613 30591 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:34.331300 31151 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.342s	user 0.001s	sys 0.000s
I20240426 13:21:34.330402 31024 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:34.331564 31027 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:34.362362 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:34.371781 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 168286us
I20240426 13:21:34.336762 30778 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:34.372196 25102 raft_consensus.cc:2362] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19. Re-sending same reply.
I20240426 13:21:34.311126 31013 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.378731 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:34.383064 25102 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:34.358422 31006 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:34.332979 30440 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.394634 31126 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:34.402665 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 234041us
I20240426 13:21:34.404255 25026 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:34.350411 31001 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.334398 31118 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.327414 31167 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.370877 30591 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.425627 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:34.390746 31006 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:34.429796 25022 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:34.433974 25023 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20240426 13:21:34.438431 31013 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:34.376643 31024 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.444228 25102 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:34.445708 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:34.447751 25025 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:34.443586 25096 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:34.371245 31168 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
W20240426 13:21:34.428692 31142 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.567s	user 0.003s	sys 0.006s
I20240426 13:21:34.451464 25101 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:34.463428 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:34.463582 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:34.426018 31169 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:34.498857 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:34.499598 25023 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:34.494257 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:34.442240 31006 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:34.376041 31175 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20240426 13:21:34.506146 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:34.508612 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 337764us
I20240426 13:21:34.508781 25102 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:34.513298 24953 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 15
I20240426 13:21:34.520638 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:34.522327 25619 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:34.522316 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:34.503651 30977 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
W20240426 13:21:34.524703 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 149387us
I20240426 13:21:34.573365 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.579258 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:34.592615 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:34.569635 25027 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:34.519591 30974 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:34.510948 31171 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
I20240426 13:21:34.450569 31133 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.504698 31169 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.519717 31188 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:34.533432 30811 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.894s	user 0.008s	sys 0.004s
I20240426 13:21:34.611382 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.595928 25023 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:34.775892 31165 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:34.775342 30811 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.649895 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:34.777297 31152 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.876s	user 0.025s	sys 0.062s
W20240426 13:21:34.777704 31152 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.025s	sys 0.062s
I20240426 13:21:34.646296 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2686) took 1260 ms. Trace:
I20240426 13:21:34.616252 31164 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:21:34.677656 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:34.615497 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:34.781939 25618 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:34.640832 31024 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:34.779704 31188 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.597026 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2458) took 1035 ms. Trace:
I20240426 13:21:34.654672 31108 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.798609 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1110) took 1186 ms. Trace:
I20240426 13:21:34.826598 24956 rpcz_store.cc:276] 0426 13:21:33.539199 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.545065 (+  5866us) service_pool.cc:225] Handling call
0426 13:21:34.529683 (+984618us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54000bfc54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.574139 (+ 44456us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541266304}
I20240426 13:21:34.793363 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2680) took 1408 ms. Trace:
I20240426 13:21:34.830778 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2340) took 1061 ms. Trace:
I20240426 13:21:34.605641 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:34.796844 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.831493 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:34.831898 24952 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:34.790061 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3136) took 1094 ms. Trace:
I20240426 13:21:34.645572 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:34.832062 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 226458us
I20240426 13:21:34.585757 30977 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:34.794291 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:34.791126 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:34.831396 25031 rpcz_store.cc:276] 0426 13:21:33.769032 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.770014 (+   982us) service_pool.cc:225] Handling call
0426 13:21:34.830588 (+1060574us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54002625d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.830704 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522691968}
I20240426 13:21:34.645509 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2687) took 1259 ms. Trace:
I20240426 13:21:34.809002 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.809644 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 1108) took 1263 ms. Trace:
I20240426 13:21:34.841790 24949 rpcz_store.cc:276] 0426 13:21:33.545762 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.548161 (+  2399us) service_pool.cc:225] Handling call
0426 13:21:34.809448 (+1261287us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.809592 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1647705984}
I20240426 13:21:34.701344 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:34.646876 25105 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20240426 13:21:34.845674 25105 pending_rounds.cc:76] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 14
I20240426 13:21:34.846077 25105 pending_rounds.cc:97] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20240426 13:21:34.846577 25105 raft_consensus.cc:2866] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:34.805692 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2311) took 1036 ms. Trace:
I20240426 13:21:34.847085 25096 rpcz_store.cc:276] 0426 13:21:33.768989 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.779572 (+ 10583us) service_pool.cc:225] Handling call
0426 13:21:34.805490 (+1025918us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.805627 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1256798592}
I20240426 13:21:34.721359 31040 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:34.777662 25028 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:34.853035 31040 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.854133 30977 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.855147 31040 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:34.644716 30602 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:34.855824 30602 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:34.841027 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:34.856669 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:34.856984 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.858928 31118 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:34.860087 31164 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:34.860415 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:34.861277 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:34.651536 30874 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.863014 24948 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:34.639593 31090 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:21:34.641867 30591 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:34.864326 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70112.9us
I20240426 13:21:34.865610 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.865772 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.831053 25103 rpcz_store.cc:276] 0426 13:21:33.384547 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.641470 (+256923us) service_pool.cc:225] Handling call
0426 13:21:33.641683 (+   213us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:34.032709 (+391026us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:34.793137 (+760428us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54003edad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:34.793297 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":897913088}
I20240426 13:21:34.832841 25026 rpcz_store.cc:276] 0426 13:21:33.695896 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.698293 (+  2397us) service_pool.cc:225] Handling call
0426 13:21:34.789913 (+1091620us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54002f9c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.790009 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1428758400}
W20240426 13:21:34.809705 31192 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20240426 13:21:34.649278 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:34.614578 30760 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:34.662050 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:34.646070 31184 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20240426 13:21:34.779377 25098 rpcz_store.cc:276] 0426 13:21:33.386267 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.549819 (+163552us) service_pool.cc:225] Handling call
0426 13:21:34.646046 (+1096227us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b5400169754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.646236 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1533942656}
I20240426 13:21:34.878255 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:34.879735 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 101248us
I20240426 13:21:34.879963 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3140) took 1089 ms. Trace:
I20240426 13:21:34.880240 25098 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:34.880383 25027 rpcz_store.cc:276] 0426 13:21:33.790270 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.790899 (+   629us) service_pool.cc:225] Handling call
0426 13:21:34.879759 (+1088860us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.879910 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":964467584}
I20240426 13:21:34.850474 31160 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:34.643038 31167 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:34.851729 31179 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:34.894246 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:34.894845 25098 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:34.864615 31089 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:34.616130 30778 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:34.861032 31164 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:21:34.799422 30665 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.618s	user 0.007s	sys 0.003s
W20240426 13:21:34.803066 31196 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20240426 13:21:34.801492 31183 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.191s	user 0.002s	sys 0.000s
I20240426 13:21:34.837828 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:34.717700 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:34.867499 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:34.868373 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:34.867295 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:34.888705 31001 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:34.907811 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:34.827734 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 216420us
I20240426 13:21:34.871397 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.605079 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.794154 31129 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:34.830739 24951 rpcz_store.cc:276] 0426 13:21:33.612000 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.612145 (+   145us) service_pool.cc:225] Handling call
0426 13:21:33.612483 (+   338us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:33.612822 (+   339us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:33.613101 (+   279us) log.cc:842] Serialized 14 byte log entry
0426 13:21:33.760355 (+147254us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:33.774614 (+ 14259us) log.cc:842] Serialized 26 byte log entry
0426 13:21:33.774768 (+   154us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:21:33.775449 (+   681us) log.cc:842] Serialized 14 byte log entry
0426 13:21:33.777232 (+  1783us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:33.777295 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:33.853605 (+ 76310us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b18002fc980. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:33.861624 (+  8019us) spinlock_profiling.cc:243] Waited 4.25 ms on lock 0x7b18002fc980. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:34.397446 (+535822us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:34.435707 (+ 38261us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b18002fc980. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:34.798408 (+362701us) raft_consensus.cc:1671] finished
0426 13:21:34.798429 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:34.798538 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59845,"prepare.queue_time_us":1176,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":10151,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":15467,"spinlock_wait_cycles":37570304,"thread_start_us":146833,"threads_started":1,"wal-append.queue_time_us":169479}
W20240426 13:21:34.651486 31034 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.001s
I20240426 13:21:34.802240 31006 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:34.915020 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.790294 31188 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:34.848897 30668 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:34.830607 30811 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:34.849450 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:34.645418 30440 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:34.841810 25101 rpcz_store.cc:276] 0426 13:21:33.386511 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.529952 (+143441us) service_pool.cc:225] Handling call
0426 13:21:34.645287 (+1115335us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:34.645448 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1915539840}
I20240426 13:21:34.924182 25096 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:34.882673 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:34.879452 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:34.925143 25103 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:34.888444 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:34.926990 24956 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:34.896334 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:34.864398 31090 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:34.900154 25102 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:34.928619 24949 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:34.875667 31121 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:34.905525 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:34.910208 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.912678 31040 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20240426 13:21:34.917603 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49439.4us
W20240426 13:21:34.908814 31189 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.325s	user 0.002s	sys 0.000s
I20240426 13:21:34.889245 30591 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20240426 13:21:34.902163 31179 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:34.895462 31158 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20240426 13:21:34.906205 31089 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:34.922207 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.946069 24955 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:34.956269 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:34.952718 25615 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:34.914032 31129 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:34.916788 31010 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:34.917062 31024 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:34.944698 31188 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:34.956826 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.894490 31160 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.935081 31001 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:34.905956 30665 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:34.944511 31034 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:34.995944 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.997392 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.004637 31024 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:35.004787 30665 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:34.998275 31001 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.005656 31001 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:35.006834 25097 raft_consensus.cc:1203] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.8->[16.9-20.11]   Dedup: 16.9->[18.10-20.11]
I20240426 13:21:35.007292 25097 pending_rounds.cc:76] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 9
I20240426 13:21:35.007333 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 828) took 1134 ms. Trace:
I20240426 13:21:35.007666 25097 pending_rounds.cc:97] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20240426 13:21:35.008085 25097 raft_consensus.cc:2866] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:35.005664 31024 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.008229 24950 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:34.935675 31165 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:35.017107 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.021373 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.021914 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:34.961581 31171 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:34.996742 31188 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.007900 25623 rpcz_store.cc:276] 0426 13:21:33.873304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.873459 (+   155us) service_pool.cc:225] Handling call
0426 13:21:33.873703 (+   244us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:33.884001 (+ 10298us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:33.884028 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:33.899384 (+ 15356us) log.cc:842] Serialized 26 byte log entry
0426 13:21:34.338350 (+438966us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:34.338376 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:34.338421 (+    45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:34.800871 (+462450us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:35.000370 (+199499us) raft_consensus.cc:1671] finished
0426 13:21:35.000398 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:35.007269 (+  6871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2199,"raft.queue_time_us":12162,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":13556,"spinlock_wait_cycles":11737472,"thread_start_us":401822,"threads_started":1,"wal-append.queue_time_us":593900}
W20240426 13:21:34.998795 31186 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.414s	user 0.003s	sys 0.000s
I20240426 13:21:34.998239 30707 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:35.025867 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:34.948319 30668 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:35.028347 24947 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:35.028365 25024 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:21:35.045557 24956 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:35.046308 31115 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:35.046805 31115 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.047147 30668 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.047997 31006 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.049201 25096 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:35.050134 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.050318 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.051393 25104 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:35.032456 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2467) took 1132 ms. Trace:
I20240426 13:21:35.057060 25096 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:35.056950 24950 rpcz_store.cc:276] 0426 13:21:33.900461 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.935339 (+ 34878us) service_pool.cc:225] Handling call
0426 13:21:33.935565 (+   226us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:35.012902 (+1077337us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:35.032245 (+ 19343us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:35.032397 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2466455936}
I20240426 13:21:35.041539 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.044064 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.060405 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:35.058678 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:35.013262 31145 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:35.068728 24950 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:35.061924 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.069911 25099 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:35.070889 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.015554 30665 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:35.045558 25102 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:35.062791 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.065152 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.075196 25623 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15.
I20240426 13:21:35.065770 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.025002 30602 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.080415 25099 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:35.081725 24955 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:35.073123 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.017966 31169 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.076272 24920 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15.
I20240426 13:21:35.089931 24920 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20240426 13:21:35.041326 31188 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.097651 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:35.042174 31010 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:35.067193 31139 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.200s	user 0.005s	sys 0.028s
I20240426 13:21:34.953011 31102 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
I20240426 13:21:35.005810 31034 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.064169 31170 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:35.025601 30315 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20240426 13:21:35.081668 31173 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:21:35.124534 31202 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20240426 13:21:35.128265 24952 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:35.128960 31165 raft_consensus.cc:2741] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20240426 13:21:35.115267 31010 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.147514 31160 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.148391 31160 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:35.122078 31153 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20240426 13:21:35.150260 31185 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:21:35.150513 31153 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.151058 31153 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.080533 31089 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.153901 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3131) took 1787 ms. Trace:
I20240426 13:21:35.157327 30602 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.159178 31169 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20240426 13:21:35.166711 31205 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:21:35.168723 31115 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.169739 31090 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:35.171353 31090 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:35.171979 31108 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.172784 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.173501 25031 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.173522 31155 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.053s
I20240426 13:21:35.175143 25070 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.175443 25070 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20240426 13:21:35.066232 31006 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:35.176100 31108 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.176462 31006 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.178215 31006 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.083519 31179 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.179831 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:35.065280 30974 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:35.043396 31158 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.077s
I20240426 13:21:35.081933 30665 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.118676 25099 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:35.154439 25029 rpcz_store.cc:276] 0426 13:21:33.366055 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.480953 (+114898us) service_pool.cc:225] Handling call
0426 13:21:33.483928 (+  2975us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:33.530001 (+ 46073us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:33.530029 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:33.550563 (+ 20534us) log.cc:842] Serialized 26 byte log entry
0426 13:21:33.781445 (+230882us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:33.781499 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:34.825634 (+1044135us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:34.825702 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:34.900159 (+ 74457us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:35.153627 (+253468us) raft_consensus.cc:1671] finished
0426 13:21:35.153658 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:35.153829 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8027,"raft.queue_time_us":8115,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":71,"spinlock_wait_cycles":104398976,"thread_start_us":230506,"threads_started":1,"wal-append.queue_time_us":241353}
W20240426 13:21:35.195153 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2314) took 1425 ms (client timeout 1816 ms). Trace:
I20240426 13:21:35.120927 31170 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:35.195778 25104 rpcz_store.cc:269] 0426 13:21:33.769990 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.779014 (+  9024us) service_pool.cc:225] Handling call
0426 13:21:35.194971 (+1415957us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400168d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:35.195088 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2534034688}
I20240426 13:21:35.204741 25104 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20240426 13:21:35.205374 25100 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:35.207126 25104 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20240426 13:21:35.207801 25104 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 10
I20240426 13:21:35.208335 25104 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20240426 13:21:35.210034 25104 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:35.191664 30665 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.194501 30974 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:35.208704 31170 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:35.076558 31129 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.210350 31145 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.246881 31124 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.211724 31001 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.265906 25029 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:35.265828 31124 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.275328 31040 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:35.275703 31040 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:35.276542 31040 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.278095 31040 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:35.287757 31010 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.288540 30811 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.289058 30811 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.290938 30811 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.295794 30974 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.298997 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.299738 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:35.302569 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.256572 31145 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:35.331307 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 215860us
W20240426 13:21:35.333174 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 235579us
W20240426 13:21:35.335372 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 213492us
I20240426 13:21:35.339931 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.341099 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.344512 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.349032 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.351087 30315 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:35.351498 30315 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:35.352294 30315 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.353506 30315 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:35.330994 31010 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:35.353991 31124 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.354091 31010 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.354533 31010 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:35.355316 31001 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.329471 31117 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.359465 25104 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:35.362258 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:35.359503 25102 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:35.314792 30811 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.368357 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.332229 30440 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:35.378654 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.328680 31108 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20240426 13:21:35.358402 31101 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:21:35.433229 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 357109us
I20240426 13:21:35.430065 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:35.434700 25623 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.431488 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.436152 25027 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:35.432056 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.432271 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.437502 24952 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
W20240426 13:21:35.440321 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 70220.1us
W20240426 13:21:35.443688 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106107us
I20240426 13:21:35.365015 31158 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:35.437536 24955 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:35.446326 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.449661 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.451776 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.352602 31145 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.455797 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.461704 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2321) took 1094 ms. Trace:
I20240426 13:21:35.460917 25069 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.436388 30760 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.446578 30707 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.370950 31117 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.462257 25101 rpcz_store.cc:276] 0426 13:21:34.367259 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:34.924519 (+557260us) service_pool.cc:225] Handling call
0426 13:21:34.949797 (+ 25278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:34.950057 (+   260us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:35.231084 (+281027us) log.cc:842] Serialized 14 byte log entry
0426 13:21:35.461288 (+230204us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:35.461355 (+    67us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:35.461426 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:35.461508 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:35.461639 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41071,"prepare.queue_time_us":41619,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"thread_start_us":453090,"threads_started":2}
I20240426 13:21:35.376302 30665 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.447731 31133 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15.
I20240426 13:21:35.451879 31158 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.443542 31101 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:35.485145 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.485888 24956 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 12: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:35.486752 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.376461 30811 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.481844 30707 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.493331 31034 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.493630 30811 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.494838 30707 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.495643 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2694) took 1970 ms. Trace:
I20240426 13:21:35.496006 25105 rpcz_store.cc:276] 0426 13:21:33.525458 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:33.694711 (+169253us) service_pool.cc:225] Handling call
0426 13:21:34.646453 (+951742us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:34.850508 (+204055us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:35.495469 (+644961us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b5400169750. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:35.495592 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2145541504}
I20240426 13:21:35.500187 30874 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:35.502547 31010 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:35.505338 31208 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:21:35.506069 24951 raft_consensus.cc:2354] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20.
I20240426 13:21:35.506601 30665 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.507747 31212 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:35.470543 31024 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:35.510982 31024 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.511317 31024 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.511579 31158 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.512118 31101 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:35.513789 24948 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.446921 31173 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:35.514541 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.483680 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.460047 31078 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.477769 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.491765 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.491887 24953 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:35.478017 30760 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.518400 31078 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.519620 30760 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.521157 31173 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.523337 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:35.536976 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59271.1us
I20240426 13:21:35.544850 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.499642 31153 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.562140 25069 leader_election.cc:398] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20.
I20240426 13:21:35.482602 31169 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:35.562637 25069 leader_election.cc:401] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20240426 13:21:35.567198 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93770.9us
I20240426 13:21:35.568558 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.505357 31117 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.570902 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.571624 25027 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
W20240426 13:21:35.571717 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 80573.4us
I20240426 13:21:35.572363 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.573078 24955 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:35.571897 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.577629 31117 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.580677 31117 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.575369 31169 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:35.582453 31117 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.568914 31161 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.577751 31078 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.583173 31118 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.563295 31197 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:35.583979 31197 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:35.584849 31197 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.583634 31118 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.575424 31090 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.589838 31118 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.590400 31118 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.591156 31101 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.592146 31010 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.580107 31153 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.581991 31169 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.565763 30707 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.583060 31161 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.596390 30707 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:35.597029 31219 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:21:35.598008 31161 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.598403 30707 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.590793 31118 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.599157 31034 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:35.598858 31217 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20240426 13:21:35.599987 31216 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20240426 13:21:35.601480 31215 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.097s	user 0.001s	sys 0.001s
I20240426 13:21:35.602272 31168 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.602816 31168 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:21:35.604450 31218 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20240426 13:21:35.604835 31168 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.605022 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.605515 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:35.606315 31213 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20240426 13:21:35.608266 31206 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20240426 13:21:35.601626 31118 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.608847 31034 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:35.609216 31034 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.609526 31034 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:35.605604 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
W20240426 13:21:35.612224 31204 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.106s	user 0.002s	sys 0.000s
I20240426 13:21:35.612752 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.613301 31101 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.613801 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2320) took 1247 ms. Trace:
I20240426 13:21:35.614005 30707 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:35.614379 25031 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:35.614470 30707 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.615028 30707 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:35.616847 24947 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:35.617226 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:35.614204 25098 rpcz_store.cc:276] 0426 13:21:34.366680 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:34.900738 (+534058us) service_pool.cc:225] Handling call
0426 13:21:34.900952 (+   214us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:34.901215 (+   263us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:34.901234 (+    19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:34.906765 (+  5531us) log.cc:842] Serialized 26 byte log entry
0426 13:21:35.445006 (+538241us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:35.469571 (+ 24565us) log.cc:842] Serialized 14 byte log entry
0426 13:21:35.469736 (+   165us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:35.469804 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:35.611646 (+141842us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:35.613540 (+  1894us) raft_consensus.cc:1671] finished
0426 13:21:35.613631 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:35.613749 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24907,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":3409,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":5817,"spinlock_wait_cycles":51483392,"thread_start_us":499861,"threads_started":1,"wal-append.queue_time_us":588100}
I20240426 13:21:35.621490 24955 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:35.625159 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.625517 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:35.629256 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.630451 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.629362 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.626088 24955 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:35.631129 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.631855 25103 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:21:35.631903 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.633832 25028 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20240426 13:21:35.633790 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.634130 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.634584 25022 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.634816 25102 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:35.636514 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.637610 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.639518 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.640079 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2323) took 1271 ms. Trace:
I20240426 13:21:35.641985 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:35.595890 31214 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20240426 13:21:35.642575 25024 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:35.643648 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.644914 25099 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15.
I20240426 13:21:35.644297 24954 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:35.640499 25097 rpcz_store.cc:276] 0426 13:21:34.368713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:34.992683 (+623970us) service_pool.cc:225] Handling call
0426 13:21:35.006648 (+ 13965us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:21:35.008913 (+  2265us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:35.009182 (+   269us) log.cc:842] Serialized 14 byte log entry
0426 13:21:35.371407 (+362225us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:35.373904 (+  2497us) log.cc:842] Serialized 52 byte log entry
0426 13:21:35.438439 (+ 64535us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:35.438728 (+   289us) log.cc:842] Serialized 14 byte log entry
0426 13:21:35.470027 (+ 31299us) log.cc:842] Serialized 14 byte log entry
0426 13:21:35.587587 (+117560us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b180047c260. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:35.588191 (+   604us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b180047c260. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:35.600966 (+ 12775us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:35.639735 (+ 38769us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:35.639807 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:35.639875 (+    68us) raft_consensus.cc:1671] finished
0426 13:21:35.639889 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:35.640013 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37590,"prepare.queue_time_us":38721,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":88,"raft.queue_time_us":17848,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":50014,"spinlock_wait_cycles":68532864,"thread_start_us":531289,"threads_started":2,"wal-append.queue_time_us":494977}
I20240426 13:21:35.641384 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.649947 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.650583 25027 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 18
W20240426 13:21:35.654449 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84152.5us
I20240426 13:21:35.654563 31211 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:35.659932 31211 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:35.660313 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.660434 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.660847 31211 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.661414 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.662019 25105 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:35.643924 31010 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20240426 13:21:35.627599 31040 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:35.646991 31188 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.001s
I20240426 13:21:35.668509 31188 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.668926 24955 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20240426 13:21:35.669605 24955 pending_rounds.cc:76] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:21:35.670037 24955 pending_rounds.cc:97] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20240426 13:21:35.670497 24955 raft_consensus.cc:2866] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:35.575933 31193 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20240426 13:21:35.656129 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.680284 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.686420 25026 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:21:35.686928 25027 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:35.688892 25069 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:35.690063 25022 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:35.690897 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:35.691642 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52860.8us
I20240426 13:21:35.698495 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.670651 31188 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:35.702646 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.654935 31108 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:35.704154 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.706842 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.710201 30707 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.697568 31040 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:35.712496 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.719583 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.719899 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.720578 25022 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:35.700349 30614 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:35.727417 31034 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.736301 25099 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240426 13:21:35.723768 31133 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:35.702806 31090 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20.
I20240426 13:21:35.726872 31024 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.728564 24951 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240426 13:21:35.723677 30672 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:21:35.730593 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.731199 25100 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:21:35.731751 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.732875 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:35.734267 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.749882 25098 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:35.749994 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.750737 25100 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.734987 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.751442 25103 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:35.735133 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.735368 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.735013 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:35.738457 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.750944 25098 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:21:35.753484 24953 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:35.753146 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:35.659219 31170 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:35.718658 30707 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.755703 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:35.729118 31221 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20240426 13:21:35.695767 31169 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20240426 13:21:35.781733 31024 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.705789 31153 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:35.719352 30977 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:35.747371 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:35.754482 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.748667 24948 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:35.801214 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.760906 24951 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:21:35.762352 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:35.770740 31227 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:35.802817 25027 raft_consensus.cc:2396] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 20 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
W20240426 13:21:35.802820 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48393.2us
I20240426 13:21:35.806608 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.807374 25022 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:35.812456 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:35.789831 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.771782 31034 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:35.772248 31197 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:35.772735 30760 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:35.730327 31167 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:35.718585 31193 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20240426 13:21:35.720698 31108 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:35.729000 31117 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20240426 13:21:35.838150 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:35.774112 31170 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:35.774271 31182 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:35.788749 25100 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:35.846928 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:35.791328 24950 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:35.798090 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:35.801903 24954 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:21:35.800911 25099 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:35.796538 24953 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:35.819396 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.821264 24952 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:35.852347 24921 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:21:35.852640 24921 leader_election.cc:401] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:35.821722 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:35.824590 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.826038 25618 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:21:35.853281 25620 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:35.829962 31024 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:35.830324 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.823122 25022 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:35.822530 31197 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:35.842316 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.854126 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.827705 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:35.855615 31013 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:35.855859 24948 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:35.856202 31225 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:35.860618 31220 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.190s	user 0.000s	sys 0.002s
I20240426 13:21:35.862030 25099 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:21:35.865928 25025 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20240426 13:21:35.886946 25025 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 12
I20240426 13:21:35.887180 25025 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20240426 13:21:35.887411 25025 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:35.859635 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.833750 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:35.870961 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.872972 24952 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:35.874004 25026 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:35.830363 30760 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:35.883424 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55816.8us
I20240426 13:21:35.891609 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.829815 30974 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20240426 13:21:35.832263 31160 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20240426 13:21:35.892668 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.906457 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.903046 24954 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:35.912724 24950 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20240426 13:21:35.873759 31090 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:35.914631 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.917372 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.915403 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:35.883054 31225 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.889055 30843 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20240426 13:21:35.889955 31024 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:35.890726 31169 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:35.917232 31078 raft_consensus.cc:2741] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20240426 13:21:35.931622 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.932410 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:35.933715 25102 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:35.890105 31118 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:35.934284 25026 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20240426 13:21:35.935990 25096 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:35.901460 30760 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:35.903884 31170 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:35.913875 31101 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:21:35.914006 31168 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:35.938333 31117 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:35.940461 31224 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:35.930153 31225 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:35.919386 31040 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:35.919138 30672 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:35.931277 31153 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:35.925848 31013 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:35.967525 31168 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.968797 31126 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:35.967271 31117 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:35.931421 31188 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20240426 13:21:35.939617 31145 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.085s
W20240426 13:21:36.014464 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45622.5us
I20240426 13:21:35.919315 31089 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20240426 13:21:35.914014 31115 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20240426 13:21:35.969009 31102 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:35.967231 31226 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.051295 31226 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:36.015300 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 88867.1us
I20240426 13:21:35.961035 31118 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:36.052580 31118 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:36.052728 24921 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15.
I20240426 13:21:36.053767 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:35.973315 31155 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20240426 13:21:35.963604 31171 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20240426 13:21:35.940871 31164 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20240426 13:21:36.007233 31078 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.035s
I20240426 13:21:36.060761 25026 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20240426 13:21:36.031807 31117 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:36.074672 25025 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20240426 13:21:36.025094 30440 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20240426 13:21:36.030663 31168 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:21:36.027729 31224 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:36.024875 31161 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20240426 13:21:36.074326 25022 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: [14.10-17.11]
I20240426 13:21:35.937510 31197 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.063060 24951 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:36.067811 31090 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.070123 31165 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:36.079396 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:36.088292 30668 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20240426 13:21:36.108847 31238 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:21:36.110958 31224 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.119074 31090 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.132498 31090 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:36.119293 31165 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:36.133428 31165 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:36.115386 31129 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:36.065070 30672 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:36.041432 31143 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20240426 13:21:36.126472 25031 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:21:36.124011 30668 consensus_queue.cc:1225] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:21:36.128049 25022 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:36.136452 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:36.077981 31108 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:36.146961 25616 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:21:36.037652 31024 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.087467 31182 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:21:36.088042 31089 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20240426 13:21:36.162984 25097 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:21:36.072181 31209 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.084s
W20240426 13:21:36.191833 31173 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.016s	sys 0.017s
I20240426 13:21:36.159552 31108 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.048s
W20240426 13:21:36.219694 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 143473us
I20240426 13:21:36.160507 31090 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20240426 13:21:36.124639 31013 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:36.213881 31173 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.017s	sys 0.017s
I20240426 13:21:36.229383 31197 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:36.219913 31171 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.198252 30672 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.274938 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:36.234779 31153 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20240426 13:21:36.268198 31226 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:36.268810 31171 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.290050 31211 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:36.290558 31211 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:36.292460 31222 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.638s	user 0.004s	sys 0.000s
W20240426 13:21:36.295511 31241 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:21:36.271821 31173 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20240426 13:21:36.298308 31117 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.300431 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:36.329999 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64268.9us
I20240426 13:21:36.287822 31171 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:36.331957 24947 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:36.208434 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:21:36.330993 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 124283us
I20240426 13:21:36.342310 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:36.342406 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:36.291810 30760 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.376382 24956 pending_rounds.cc:76] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
W20240426 13:21:36.452279 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79989.1us
I20240426 13:21:36.458567 24956 pending_rounds.cc:97] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20240426 13:21:36.458949 24956 raft_consensus.cc:2866] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:36.344259 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:36.462960 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119718us
I20240426 13:21:36.468930 25102 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20240426 13:21:36.475220 25100 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20240426 13:21:36.481941 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:36.534853 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:36.291438 31211 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.549552 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:36.368084 30977 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.546s	user 0.007s	sys 0.003s
I20240426 13:21:36.453979 31171 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.361832 31117 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:36.509594 31089 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:36.508749 31173 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:36.566862 24947 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:21:36.551146 31010 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.553001 31224 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:36.367036 31168 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:36.562358 30977 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:21:36.567376 31171 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:36.611605 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48226us
I20240426 13:21:36.567379 31089 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:36.570698 31173 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.630897 24952 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
W20240426 13:21:36.619541 30614 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.013s	sys 0.019s
I20240426 13:21:36.561564 31240 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20240426 13:21:36.603498 31024 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.617770 31089 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:36.668072 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47758.3us
W20240426 13:21:36.658527 30614 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.013s	sys 0.019s
I20240426 13:21:36.606729 31010 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:36.675067 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 323595us
W20240426 13:21:36.696942 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 365388us
I20240426 13:21:36.720984 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:36.723245 25030 raft_consensus.cc:2354] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14.
I20240426 13:21:36.727535 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:36.691643 30760 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:36.672763 31024 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.729971 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:36.751645 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65310.4us
I20240426 13:21:36.730566 31108 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:36.746510 31118 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.693s	user 0.007s	sys 0.007s
W20240426 13:21:36.794310 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 90402.2us
W20240426 13:21:36.800462 31259 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20240426 13:21:36.766749 31246 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.808786 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:36.839237 25100 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:36.839363 25099 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:36.772071 31171 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:36.777604 31168 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:36.843706 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:36.789768 31118 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:36.845165 31129 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.000s
W20240426 13:21:36.845952 31243 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20240426 13:21:36.884678 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 123611us
I20240426 13:21:36.821938 31173 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:36.846758 31165 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.713s	user 0.004s	sys 0.007s
I20240426 13:21:36.875185 25100 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:36.745062 31253 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.264s
I20240426 13:21:36.789115 31108 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:36.888711 25026 raft_consensus.cc:2396] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:21:36.849787 31143 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.848366 31171 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.902144 24953 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
W20240426 13:21:36.891202 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 139820us
I20240426 13:21:36.902550 24953 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:36.901926 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:36.905499 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:36.894272 31252 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20240426 13:21:36.903885 24953 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:36.900962 31258 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.223s	user 0.000s	sys 0.000s
I20240426 13:21:36.917635 24921 leader_election.cc:398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14.
I20240426 13:21:36.921715 24921 leader_election.cc:401] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20240426 13:21:36.922596 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.009s	sys 0.015s
W20240426 13:21:36.923015 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.009s	sys 0.015s
I20240426 13:21:36.894021 31129 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:36.938802 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:36.909353 31135 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:36.904246 31010 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:36.931192 31261 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20240426 13:21:36.895439 31165 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.934334 31133 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:36.941377 31274 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20240426 13:21:36.958689 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:36.959311 25030 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:21:36.910562 31171 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.961546 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:36.940358 31121 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.966871 24948 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:36.906216 31143 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:36.873245 31246 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.949169 30614 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.956687 31290 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:36.975469 31270 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:36.947113 31271 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.941890 31129 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:36.985291 24950 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:36.955125 31135 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.973109 31024 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:37.073264 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 99158.6us
I20240426 13:21:36.983014 31010 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:37.032287 31225 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.007s	user 0.012s	sys 0.000s
I20240426 13:21:37.080938 31291 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
W20240426 13:21:37.081593 31264 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20240426 13:21:37.081920 31269 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:36.977303 31287 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.163s
I20240426 13:21:37.083415 31247 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.132699 31292 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:37.074312 31270 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
W20240426 13:21:37.134496 31284 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20240426 13:21:37.135152 31279 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.104s
I20240426 13:21:37.135849 31270 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:37.136703 31288 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:37.136436 31270 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:37.137116 31280 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240426 13:21:36.990478 31271 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.138710 31270 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.139016 31271 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:36.983660 31290 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:36.983407 30614 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.140807 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:37.141268 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:37.166556 25100 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:21:37.167644 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:37.092751 31118 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:36.971629 31121 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.173820 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:37.136349 31247 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:37.226328 31281 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20240426 13:21:37.227624 31247 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.228248 31143 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.229555 31171 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.230834 31129 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.235181 31121 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:37.237377 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62799.9us
I20240426 13:21:37.163831 31270 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.439720 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:37.224988 31267 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:37.225155 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:37.223574 24955 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:37.246692 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:37.224444 31013 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:37.477916 24956 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20240426 13:21:37.172477 31271 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:37.131451 31010 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.173839 31108 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.510617 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:37.125041 31225 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.242805 31290 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:37.549479 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54201.6us
I20240426 13:21:37.481894 30760 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:37.566982 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.003s	sys 0.019s
I20240426 13:21:37.567898 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:37.569087 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.003s	sys 0.019s
W20240426 13:21:37.569437 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 593835us
I20240426 13:21:37.489584 31271 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:37.570350 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 481595us
I20240426 13:21:37.573417 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:37.575827 25029 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:37.585944 24955 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
W20240426 13:21:37.511837 31224 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.031s	sys 0.043s
I20240426 13:21:37.559346 31108 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.137216 31288 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.557708 31292 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:37.613410 31224 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.031s	sys 0.043s
I20240426 13:21:37.482690 31270 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.172477 31269 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.487579 31013 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:37.608090 30672 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.012s	sys 0.062s
I20240426 13:21:37.575170 31290 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.607560 31271 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:21:37.634106 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.000s	sys 0.013s
W20240426 13:21:37.681995 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.000s	sys 0.014s
I20240426 13:21:37.685119 25105 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:37.635599 31300 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.639523 31299 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:37.644932 31293 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20240426 13:21:37.489336 31267 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.662717 31225 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:37.665199 31010 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.665834 31246 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:37.667845 31269 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.669418 31270 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:37.670977 31133 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.707s	user 0.008s	sys 0.000s
I20240426 13:21:37.672307 31126 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:37.673070 31288 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.619740 31287 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:37.674120 31089 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.004s	user 0.004s	sys 0.006s
I20240426 13:21:37.674551 31302 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:37.676149 31118 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:37.605315 30760 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.686133 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:37.779958 25105 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:37.686414 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:37.781271 25101 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:37.686993 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:37.782316 25096 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:37.680537 30672 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.012s	sys 0.063s
I20240426 13:21:37.709569 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:37.792338 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:37.761509 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:37.680784 31224 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:37.680949 31145 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:37.773572 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:37.797384 24949 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:37.774370 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:37.626905 31108 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.798727 24951 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:37.845243 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3168) took 1565 ms. Trace:
I20240426 13:21:37.845791 25031 rpcz_store.cc:276] 0426 13:21:36.280095 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.280476 (+   381us) service_pool.cc:225] Handling call
0426 13:21:36.280765 (+   289us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:36.328407 (+ 47642us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:21:36.328438 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:36.328932 (+   494us) log.cc:842] Serialized 26 byte log entry
0426 13:21:36.923428 (+594496us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:36.923486 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:36.923579 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:37.844956 (+921377us) raft_consensus.cc:1671] finished
0426 13:21:37.844986 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:37.845172 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47227,"raft.queue_time_us":49109,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":9167,"thread_start_us":578256,"threads_started":1,"wal-append.queue_time_us":792047}
W20240426 13:21:37.862990 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.002s	sys 0.027s
W20240426 13:21:37.863509 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.979s	user 0.002s	sys 0.028s
I20240426 13:21:37.778424 31133 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14.
I20240426 13:21:37.777436 31126 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:37.693486 31290 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.794979 31269 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.808070 31010 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.808518 31270 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:37.896272 25096 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:37.796144 31089 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:37.907894 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 363526us
I20240426 13:21:37.808296 31118 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.773227 31287 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.916103 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:37.766957 31299 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.885396 31225 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:37.814671 31145 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.812422 31224 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.764309 31300 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.903329 31288 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:37.933694 25104 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20240426 13:21:37.886519 31269 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:37.932938 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 353926us
I20240426 13:21:37.895578 31010 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.949671 31010 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.950789 31013 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.896009 31270 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.952778 31270 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.954371 31108 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:37.943961 30870 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:37.956413 31316 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:37.947316 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:37.955466 30870 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.866540 31302 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.960072 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:37.957284 31145 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:37.813593 31292 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.992689 25096 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:37.932505 31225 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:37.965330 31315 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
W20240426 13:21:38.006760 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.058s	user 0.000s	sys 0.017s
I20240426 13:21:38.033389 31145 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:38.033987 31089 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.034161 31270 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.034322 31145 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.034804 31270 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:38.035172 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.086s	user 0.000s	sys 0.018s
I20240426 13:21:38.036082 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:38.036554 31270 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.037068 31270 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:38.007278 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 428728us
I20240426 13:21:37.943164 31288 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:37.930984 31126 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.050311 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2508) took 1676 ms. Trace:
W20240426 13:21:38.054024 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.014s	sys 0.005s
W20240426 13:21:38.054476 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.014s	sys 0.005s
W20240426 13:21:38.054857 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 815888us
I20240426 13:21:38.050753 24952 rpcz_store.cc:276] 0426 13:21:36.374314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.374429 (+   115us) service_pool.cc:225] Handling call
0426 13:21:36.374763 (+   334us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:36.631348 (+256585us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.040392 (+1409044us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400102054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:38.050231 (+  9839us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":588398336}
I20240426 13:21:38.051710 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:38.088649 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51085.8us
I20240426 13:21:38.037508 31010 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20240426 13:21:38.088950 25096 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:38.039301 30760 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.039381 30870 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.102870 31321 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:38.109730 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.049870 31267 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.040915 31315 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:38.078840 31145 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.091388 31126 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:38.120857 25103 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:38.130504 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:38.091210 31118 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.139648 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.109344 30672 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:38.086481 31316 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20240426 13:21:38.145006 25098 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:38.146329 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:38.111572 31323 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.148697 25097 raft_consensus.cc:2354] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14.
W20240426 13:21:38.150187 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 186380us
I20240426 13:21:38.150813 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:38.128822 30870 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.151955 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.154752 24952 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:38.133751 31267 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.158027 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.157603 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:38.158856 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.138852 31290 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.167361 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.164170 24956 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:38.142375 31145 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.175504 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.175659 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.143286 31305 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.176720 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.177809 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.178969 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:38.181705 24956 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:38.182499 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:38.186370 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 226379us
I20240426 13:21:38.186960 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.189069 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.157253 30672 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:38.159938 31006 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.198726 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:38.201720 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.202845 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.129043 31321 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.202793 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:38.211982 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:38.214046 31300 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.215513 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.215939 31302 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.216650 31287 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.217561 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.218307 25615 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:38.181385 31309 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:38.175907 31224 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.191669 31118 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:38.224220 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.220378 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:38.412133 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 191989us
W20240426 13:21:38.404528 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 191692us
I20240426 13:21:38.418571 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:38.195228 31269 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.192689 30760 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:38.190762 31167 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.169116 30870 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.185787 31145 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.176138 31267 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.396628 31010 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.166636 31323 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:38.484730 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 269955us
W20240426 13:21:38.472186 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53716.6us
I20240426 13:21:38.219731 31326 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.409070 31118 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:38.425875 31302 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.453840 31155 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:38.453573 31224 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:38.455787 31269 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:38.506439 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 316100us
I20240426 13:21:38.468551 31225 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.221237 31006 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.439231 31309 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.467577 31167 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.490465 31312 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:38.523012 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.500015 31118 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:38.512809 31155 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:38.203524 31305 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.513739 31288 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.458690 30760 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.544265 24952 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 18 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20240426 13:21:38.515328 31224 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.208523 31315 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.506829 31290 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20240426 13:21:38.494997 31010 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.523061 31312 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.546097 31155 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:38.514134 31302 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.541277 31311 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:38.521631 31346 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.552201 31224 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:38.576063 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.004s	sys 0.008s
I20240426 13:21:38.522585 31269 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:38.580566 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.005s	sys 0.008s
I20240426 13:21:38.507292 30672 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.564940 31312 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:38.500321 31326 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.564572 31169 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.573755 31290 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.573597 31348 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:38.578866 31270 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.593484 31316 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:21:38.582408 31327 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.613060 24921 leader_election.cc:398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14.
I20240426 13:21:38.617161 31326 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.581594 31311 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:38.618009 31314 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.436s	user 0.000s	sys 0.001s
W20240426 13:21:38.618340 31333 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.397s	user 0.001s	sys 0.000s
I20240426 13:21:38.618474 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.624553 31344 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:38.626993 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:38.627341 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:38.599025 31287 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:38.595690 31325 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.606675 31255 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.728164 25097 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
W20240426 13:21:38.733244 31179 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.007s	sys 0.016s
I20240426 13:21:38.611730 31348 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:38.734570 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 158971us
I20240426 13:21:38.752122 31348 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:38.752774 31336 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.753235 31345 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.127s
W20240426 13:21:38.754499 31341 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20240426 13:21:38.755193 31323 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.725358 31327 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.626425 31316 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:38.581604 31346 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.727705 31317 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:38.758132 31302 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:38.637133 31321 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.759487 31311 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.747864 31344 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.601233 31169 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.631562 31270 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:38.615018 31290 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:38.760185 31325 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:39.150187 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.005s	sys 0.005s
W20240426 13:21:38.993957 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 362129us
I20240426 13:21:38.829319 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:39.156751 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.007s	sys 0.006s
W20240426 13:21:38.987907 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.750s (SENT) [suppressed 3 similar messages]
I20240426 13:21:39.158080 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:38.903964 24949 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.008s	sys 0.008s
W20240426 13:21:38.761346 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 134420us
I20240426 13:21:38.967195 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2758) took 2159 ms. Trace:
I20240426 13:21:38.761602 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.061344 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.081878 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3035) took 1915 ms. Trace:
I20240426 13:21:38.760340 31145 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:39.162087 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.985s	user 0.002s	sys 0.010s
W20240426 13:21:39.161710 24949 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.022s	sys 0.009s
W20240426 13:21:39.176201 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.999s	user 0.002s	sys 0.010s
I20240426 13:21:39.162145 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:38.725271 31102 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.167783 24947 rpcz_store.cc:276] 0426 13:21:37.166409 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.166639 (+   230us) service_pool.cc:225] Handling call
0426 13:21:37.173950 (+  7311us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:37.240554 (+ 66604us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.240597 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:37.241100 (+   503us) log.cc:842] Serialized 26 byte log entry
0426 13:21:37.895509 (+654409us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:21:37.895546 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:37.895615 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.081609 (+1185994us) raft_consensus.cc:1671] finished
0426 13:21:39.081635 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.081807 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66130,"raft.queue_time_us":66246,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":237012,"spinlock_wait_cycles":46720,"thread_start_us":385508,"threads_started":1,"wal-append.queue_time_us":487338}
I20240426 13:21:39.162382 25104 rpcz_store.cc:276] 0426 13:21:36.807695 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.808097 (+   402us) service_pool.cc:225] Handling call
0426 13:21:36.808370 (+   273us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:37.936691 (+1128321us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.966827 (+1030136us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540009d3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:38.967092 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2587231488}
W20240426 13:21:38.832360 31170 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.003s	sys 0.008s
W20240426 13:21:39.177163 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.01954e+06us
W20240426 13:21:39.240516 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78444.2us
W20240426 13:21:39.156014 31179 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.029s	user 0.016s	sys 0.052s
I20240426 13:21:38.903072 31167 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.274968 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.089140 30760 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.146454 31010 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:39.128880 31343 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.561s	user 0.001s	sys 0.000s
I20240426 13:21:39.298174 31309 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.299398 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2765) took 1615 ms. Trace:
I20240426 13:21:39.299914 25099 rpcz_store.cc:276] 0426 13:21:37.683952 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.684935 (+   983us) service_pool.cc:225] Handling call
0426 13:21:37.685189 (+   254us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:38.125076 (+439887us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:38.125108 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:38.125505 (+   397us) log.cc:842] Serialized 26 byte log entry
0426 13:21:38.191027 (+ 65522us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:38.191056 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.191117 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.299122 (+1108005us) raft_consensus.cc:1671] finished
0426 13:21:39.299151 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.299327 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":123226,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":51938,"thread_start_us":504684,"threads_started":2,"wal-append.queue_time_us":419305}
I20240426 13:21:39.301951 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.155586 31352 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:39.149906 31310 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:39.051127 31013 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:39.248212 31349 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:39.249192 31347 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
W20240426 13:21:39.249804 31342 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20240426 13:21:39.051635 31292 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:39.308724 31118 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.763s	user 0.009s	sys 0.001s
I20240426 13:21:38.825856 31326 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:39.159667 31269 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:38.888056 31225 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:38.753271 31305 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.876853 31288 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.292210 31316 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:39.352514 31346 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.352932 31316 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:21:39.353672 31364 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:21:39.355032 31270 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.348997 31225 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.089774 31155 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.356091 31290 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.357205 31225 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:39.357806 31225 consensus_queue.cc:1159] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 10 status { last_received { term: 10 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 10 index: 8 } }
I20240426 13:21:39.358408 31327 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:39.240355 31170 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.119s	user 0.017s	sys 0.014s
I20240426 13:21:39.342541 31345 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.359212 31170 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:39.297398 31312 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.694s	user 0.006s	sys 0.001s
I20240426 13:21:39.241567 31321 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:39.368358 31224 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.775s	user 0.004s	sys 0.005s
I20240426 13:21:39.364326 31288 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.368695 31321 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.368335 31305 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:39.369895 31305 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:39.320261 31349 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.370651 31321 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.223737 31317 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.251983 31323 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.371672 31323 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.369972 31288 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:39.050523 31267 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:38.747452 31255 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.371286 31317 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.374086 31255 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.312317 31388 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.359687 31345 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.374948 31388 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.376521 31345 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.376825 31388 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.323895 31292 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.350775 31325 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.352293 31339 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:39.377586 31344 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.377696 31339 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.243551 31336 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.358211 31290 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:39.176584 31373 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.320504 31352 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.371919 31349 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.371320 31305 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.274791 31302 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.373687 31348 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.340523 31326 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.382838 31373 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.387100 31336 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.387466 31352 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:39.387512 31326 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.388056 31373 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.384825 31302 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.385147 31323 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.390188 31348 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.390373 31302 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:39.380245 31312 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.392617 31312 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:39.389111 31339 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:39.431450 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.034s
W20240426 13:21:39.432242 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.011s	sys 0.034s
W20240426 13:21:39.250988 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93826.8us
I20240426 13:21:39.437290 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3027) took 3162 ms. Trace:
I20240426 13:21:39.438482 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.440215 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2525) took 1285 ms. Trace:
I20240426 13:21:39.440311 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3043) took 1031 ms. Trace:
I20240426 13:21:39.440876 24947 rpcz_store.cc:276] 0426 13:21:38.408896 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.273930 (+865034us) service_pool.cc:225] Handling call
0426 13:21:39.440217 (+166287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.441637 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3040) took 1253 ms. Trace:
I20240426 13:21:39.442703 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.442108 24956 rpcz_store.cc:276] 0426 13:21:38.187662 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.400176 (+212514us) service_pool.cc:225] Handling call
0426 13:21:38.400497 (+   321us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:38.441593 (+ 41096us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:38.456305 (+ 14712us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:38.489251 (+ 32946us) log.cc:842] Serialized 52 byte log entry
0426 13:21:38.693454 (+204203us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:38.693512 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.693610 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.441070 (+747460us) raft_consensus.cc:1671] finished
0426 13:21:39.441106 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.441426 (+   320us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22304,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":117609,"threads_started":1,"wal-append.queue_time_us":750912}
I20240426 13:21:39.438517 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:39.446030 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.448896 24956 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:39.437924 24954 rpcz_store.cc:276] 0426 13:21:36.275071 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.276225 (+  1154us) service_pool.cc:225] Handling call
0426 13:21:36.276605 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:36.330196 (+ 53591us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:36.330225 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:36.335107 (+  4882us) log.cc:842] Serialized 26 byte log entry
0426 13:21:38.591958 (+2256851us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b18004a29c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:39.008265 (+416307us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:39.008318 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.008412 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.436286 (+427874us) raft_consensus.cc:1671] finished
0426 13:21:39.436373 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.436982 (+   609us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53229,"raft.queue_time_us":58379,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":4402,"spinlock_wait_cycles":28270976,"thread_start_us":416091,"threads_started":1,"wal-append.queue_time_us":2964357}
I20240426 13:21:39.464996 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.466094 24954 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:39.451910 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.467428 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.454854 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.454967 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.455216 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.468878 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:39.456768 25029 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20240426 13:21:39.440768 24949 rpcz_store.cc:276] 0426 13:21:38.154530 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.155469 (+   939us) service_pool.cc:225] Handling call
0426 13:21:38.179453 (+ 23984us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:38.179683 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:39.176675 (+996992us) log.cc:842] Serialized 14 byte log entry
0426 13:21:39.439331 (+262656us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.439460 (+   129us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:39.439617 (+   157us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.439753 (+   136us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.440010 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":996964,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"thread_start_us":1204703,"threads_started":2,"wal-append.queue_time_us":156207}
I20240426 13:21:39.468969 25030 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:39.447728 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:39.470206 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.470382 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.479372 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.470268 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.479656 25029 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:39.479777 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.480852 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3044) took 1038 ms. Trace:
I20240426 13:21:39.480918 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.481370 24956 rpcz_store.cc:276] 0426 13:21:38.442467 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.479383 (+1036916us) service_pool.cc:225] Handling call
0426 13:21:39.480760 (+  1377us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.481837 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.482242 25029 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:39.482488 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.480952 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.483954 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.484449 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.486613 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.497424 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.497828 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.473789 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.498926 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.500361 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.500689 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.496712 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:21:39.502211 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.504997 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.502242 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.502804 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:39.488866 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:39.506402 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.003s	sys 0.020s
W20240426 13:21:39.506827 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.003s	sys 0.020s
I20240426 13:21:39.507107 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:39.503408 25029 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:39.306093 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.512277 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.513084 25099 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:39.252519 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.516275 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:39.517122 25104 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:39.514252 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.520764 25099 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:39.521471 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.524996 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:39.508131 24949 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:39.526043 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.276162 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.528601 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:21:39.530103 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.530880 25615 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:21:38.988602 31006 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.291518 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.552502 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:39.553439 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2511) took 3178 ms. Trace:
I20240426 13:21:39.553999 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.554857 31305 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:39.555595 31309 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:39.521766 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:38.761762 30870 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.574947 24954 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:39.576259 31290 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.577080 31290 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.577843 31352 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:39.247056 31102 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.274700 31145 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:39.594380 31145 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.595005 31145 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:39.357473 31155 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.368784 31224 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.358597 31225 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.599000 31225 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:39.600421 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90683.3us
I20240426 13:21:39.600873 31312 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.601536 31225 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.601778 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.602246 31225 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:39.602968 31225 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:21:39.299309 31271 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.593s	user 0.014s	sys 0.000s
I20240426 13:21:38.746939 31287 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.532997 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.526132 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.605731 25615 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
W20240426 13:21:39.606338 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80466.6us
I20240426 13:21:39.606837 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.607153 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:39.378582 31292 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.532379 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.556404 31311 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:39.608696 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2373) took 1003 ms. Trace:
I20240426 13:21:39.608937 31393 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:39.609125 25097 rpcz_store.cc:276] 0426 13:21:38.605217 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.177995 (+572778us) service_pool.cc:225] Handling call
0426 13:21:39.289049 (+111054us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:39.506738 (+217689us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.506788 (+    50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:39.507673 (+   885us) log.cc:842] Serialized 26 byte log entry
0426 13:21:39.545824 (+ 38151us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:39.545843 (+    19us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.557634 (+ 11791us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.608208 (+ 50574us) raft_consensus.cc:1671] finished
0426 13:21:39.608237 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.608425 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8915,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":254756,"threads_started":2,"wal-append.queue_time_us":94830}
I20240426 13:21:39.611364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:39.559283 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 874) took 1991 ms (client timeout 1750 ms). Trace:
I20240426 13:21:39.161216 30668 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:39.612213 25025 rpcz_store.cc:269] 0426 13:21:37.567393 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.567577 (+   184us) service_pool.cc:225] Handling call
0426 13:21:39.558845 (+1991268us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400415fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.559069 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2438467328}
I20240426 13:21:39.612994 30668 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.614027 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.615090 25025 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:39.616756 24956 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 21 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20240426 13:21:39.534423 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:39.617584 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 83474.1us
W20240426 13:21:39.564217 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3163) took 3483 ms (client timeout 1628 ms). Trace:
I20240426 13:21:39.617995 31394 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:39.618577 31394 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:39.618435 25026 rpcz_store.cc:269] 0426 13:21:36.076786 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.076970 (+   184us) service_pool.cc:225] Handling call
0426 13:21:39.560158 (+3483188us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54002add54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.560429 (+   271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1757051648}
I20240426 13:21:39.619100 31394 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:39.619921 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.531998 30811 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.623904 30811 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.397264 31317 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.626400 30811 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.328358 31118 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.565963 31169 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.566243 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.628141 31317 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.628436 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:39.629307 31326 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.629514 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:39.630007 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 641935us
I20240426 13:21:39.630131 31326 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.630942 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:39.631629 31126 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.492s	user 0.009s	sys 0.010s
I20240426 13:21:39.583451 31323 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:39.568434 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.359699 31170 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.553995 24948 rpcz_store.cc:276] 0426 13:21:36.375152 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.375280 (+   128us) service_pool.cc:225] Handling call
0426 13:21:36.507790 (+132510us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:36.968653 (+460863us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.552845 (+2584192us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b5400161f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.553200 (+   355us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1055354624}
I20240426 13:21:39.578570 31352 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:39.586385 25104 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
W20240426 13:21:39.534813 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49215us
I20240426 13:21:39.591409 31339 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.592223 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:39.602059 31359 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.604552 31271 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.636150 25621 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:39.608727 31292 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:39.601567 31312 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.643064 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3052) took 1026 ms. Trace:
I20240426 13:21:39.617519 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.576254 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.555577 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.642343 25623 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:39.639657 25024 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:39.631994 25099 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 21 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:21:39.606837 31287 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.636394 25028 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:39.645871 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.651939 24947 rpcz_store.cc:276] 0426 13:21:38.616251 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.442320 (+826069us) service_pool.cc:225] Handling call
0426 13:21:39.642892 (+200572us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b540038efd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.643003 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326852992}
I20240426 13:21:39.648468 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.644212 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:39.639457 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71869.5us
W20240426 13:21:39.644595 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 1019) took 1469 ms (client timeout 1622 ms). Trace:
I20240426 13:21:39.680953 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.675139 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.603785 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:39.697166 25621 rpcz_store.cc:269] 0426 13:21:38.175250 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.175437 (+   187us) service_pool.cc:225] Handling call
0426 13:21:39.644320 (+1468883us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.644523 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3356622592}
I20240426 13:21:39.639662 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:39.675132 31102 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.707998 24948 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:39.709205 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.710162 24948 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:39.636231 31323 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.650627 31118 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.713032 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.714388 30668 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.714620 31321 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.715262 25025 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
I20240426 13:21:39.715341 31321 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.715852 31290 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.718171 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.720470 25026 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:39.720629 31325 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.720758 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.721462 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.721616 25615 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:39.721351 31325 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.713958 31323 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.723690 31374 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:39.724155 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.724215 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.725206 25028 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:21:39.721877 31337 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.228s	user 0.000s	sys 0.001s
I20240426 13:21:39.642330 31339 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.644160 25097 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:39.726209 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.716476 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:39.650637 31256 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.957s	user 0.009s	sys 0.026s
I20240426 13:21:39.600447 31224 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.728180 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:39.752611 24949 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:39.754946 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:39.680454 31263 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.831s	user 0.011s	sys 0.078s
I20240426 13:21:39.753582 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.648123 31292 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:39.755542 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 148561us
I20240426 13:21:39.646176 31396 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:39.756686 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.758445 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.760190 25028 raft_consensus.cc:2354] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:21:39.761134 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.762094 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:39.763454 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43931.6us
I20240426 13:21:39.762954 25622 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:39.761286 25624 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:39.724285 31374 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:39.764453 31326 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.760237 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.756783 31292 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.769541 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.767917 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3046) took 1259 ms. Trace:
I20240426 13:21:39.770727 24949 rpcz_store.cc:276] 0426 13:21:38.508639 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.755187 (+1246548us) service_pool.cc:225] Handling call
0426 13:21:39.767842 (+ 12655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.772564 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.774606 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.775316 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2529) took 1243 ms. Trace:
I20240426 13:21:39.775668 24954 rpcz_store.cc:276] 0426 13:21:38.532217 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.764658 (+1232441us) service_pool.cc:225] Handling call
0426 13:21:39.775256 (+ 10598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.769480 31170 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.777446 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2527) took 1245 ms. Trace:
I20240426 13:21:39.779542 31323 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:39.780021 31323 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:39.780423 31323 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:39.784713 31312 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.785003 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.785768 25027 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:39.786697 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.788616 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.789412 24948 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:39.777805 24954 rpcz_store.cc:276] 0426 13:21:38.531667 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.776307 (+1244640us) service_pool.cc:225] Handling call
0426 13:21:39.776807 (+   500us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.777113 (+   306us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.777142 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.777191 (+    49us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:39.777232 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.777276 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.777384 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.787758 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.796171 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.797732 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:39.805249 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.639045 31394 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.811275 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1275) took 3477 ms. Trace:
I20240426 13:21:39.812124 25621 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:39.641721 31359 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.818681 25030 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20240426 13:21:39.801354 31170 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:39.826210 31170 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.826648 31170 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:39.827853 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71075.8us
I20240426 13:21:39.828963 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:39.830420 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:39.811687 25618 rpcz_store.cc:276] 0426 13:21:36.334089 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.334213 (+   124us) service_pool.cc:225] Handling call
0426 13:21:36.334461 (+   248us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:36.361323 (+ 26862us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:36.361352 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:36.361827 (+   475us) log.cc:842] Serialized 26 byte log entry
0426 13:21:36.671976 (+310149us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:36.672018 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:36.672106 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.679441 (+3007335us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:39.791728 (+112287us) raft_consensus.cc:1671] finished
0426 13:21:39.791757 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.811088 (+ 19331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26507,"raft.queue_time_us":28349,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":11876608,"thread_start_us":309805,"threads_started":1,"wal-append.queue_time_us":413534}
I20240426 13:21:39.832515 31211 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.833915 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:39.802574 31374 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.836278 31305 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:39.836468 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1278) took 2677 ms. Trace:
I20240426 13:21:39.836863 25620 rpcz_store.cc:276] 0426 13:21:37.158918 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.159107 (+   189us) service_pool.cc:225] Handling call
0426 13:21:37.159456 (+   349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:37.236155 (+ 76699us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.236183 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:37.236224 (+    41us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:37.236252 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.836074 (+2599822us) spinlock_profiling.cc:243] Waited 76.5 ms on lock 0x7b54007521d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.836145 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.836323 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175938304}
I20240426 13:21:39.838682 31359 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.824489 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:39.839988 31271 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.837441 31326 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:39.825764 31339 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.817852 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.840751 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2769) took 2155 ms. Trace:
W20240426 13:21:39.823033 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 183585us
W20240426 13:21:39.844525 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 240932us
I20240426 13:21:39.845345 31209 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:39.845830 31209 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.845937 31371 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:39.846246 31209 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:39.846618 31371 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.835637 31374 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.844887 25100 rpcz_store.cc:276] 0426 13:21:37.684944 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.687552 (+  2608us) service_pool.cc:225] Handling call
0426 13:21:37.687943 (+   391us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:37.972370 (+284427us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.972406 (+    36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:38.003419 (+ 31013us) log.cc:842] Serialized 52 byte log entry
0426 13:21:38.140129 (+136710us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:38.140186 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.140259 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.721006 (+1580747us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:39.840232 (+119226us) raft_consensus.cc:1671] finished
0426 13:21:39.840257 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.840541 (+   284us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":188530,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":3966,"spinlock_wait_cycles":16823040,"thread_start_us":387159,"threads_started":2,"wal-append.queue_time_us":185104}
I20240426 13:21:39.851413 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 866) took 3687 ms. Trace:
I20240426 13:21:39.851697 31165 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.853539 25027 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:39.854281 25022 rpcz_store.cc:276] 0426 13:21:36.164032 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.165242 (+  1210us) service_pool.cc:225] Handling call
0426 13:21:36.165485 (+   243us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:36.197392 (+ 31907us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:36.197438 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:36.206697 (+  9259us) log.cc:842] Serialized 26 byte log entry
0426 13:21:36.513592 (+306895us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:36.513645 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:36.513734 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.649626 (+3135892us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:39.850912 (+201286us) raft_consensus.cc:1671] finished
0426 13:21:39.850939 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.851232 (+   293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31514,"raft.queue_time_us":32472,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11494,"spinlock_wait_cycles":11614592,"thread_start_us":273593,"threads_started":1,"wal-append.queue_time_us":452366}
I20240426 13:21:39.859513 24954 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20240426 13:21:39.844209 31326 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:39.859408 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.859871 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:39.836884 31305 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:39.862717 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3051) took 1248 ms. Trace:
I20240426 13:21:39.863090 24954 rpcz_store.cc:276] 0426 13:21:38.611899 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.803089 (+1191190us) service_pool.cc:225] Handling call
0426 13:21:39.853077 (+ 49988us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.859915 (+  6838us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.860128 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.860657 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1086) took 1229 ms. Trace:
I20240426 13:21:39.859692 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2374) took 1251 ms. Trace:
I20240426 13:21:39.864434 25097 rpcz_store.cc:276] 0426 13:21:38.627655 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.760003 (+1132348us) service_pool.cc:225] Handling call
0426 13:21:39.816721 (+ 56718us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.854222 (+ 37501us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.854252 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.856437 (+  2185us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:39.856479 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.856698 (+   219us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.856815 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.773658 31396 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.866369 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:39.864828 25099 rpcz_store.cc:276] 0426 13:21:38.605479 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.763743 (+1158264us) service_pool.cc:225] Handling call
0426 13:21:39.816418 (+ 52675us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.853780 (+ 37362us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.853807 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.856602 (+  2795us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:39.856626 (+    24us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.856690 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.856776 (+    86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:39.866204 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.862290 31326 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:39.872573 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48225.7us
I20240426 13:21:39.874119 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:39.874747 25028 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20240426 13:21:39.875221 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.877943 25097 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:39.902143 31247 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:39.902531 31247 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:39.902861 31247 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.903076 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:39.903930 25097 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:39.879814 25619 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
W20240426 13:21:39.904876 31403 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.269s	user 0.001s	sys 0.001s
I20240426 13:21:39.904932 24953 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.905233 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.906149 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1285) took 2039 ms. Trace:
I20240426 13:21:39.853725 31145 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:39.882213 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 1137) took 1713 ms (client timeout 1750 ms). Trace:
W20240426 13:21:39.908149 24952 rpcz_store.cc:269] 0426 13:21:38.168662 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.176493 (+  7831us) service_pool.cc:225] Handling call
0426 13:21:39.881755 (+1705262us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.882008 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":833444608}
I20240426 13:21:39.906183 25097 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:39.882541 31305 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:39.915387 24949 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20240426 13:21:39.906551 25619 rpcz_store.cc:276] 0426 13:21:37.867296 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.868681 (+  1385us) service_pool.cc:225] Handling call
0426 13:21:39.879498 (+2010817us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.904872 (+ 25374us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.905901 (+  1029us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54006b26d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.906067 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4623271808}
I20240426 13:21:39.921111 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.926146 31145 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:39.906653 31195 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:39.926084 25097 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 23: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:39.883947 31090 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:39.884737 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.885774 31325 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.883538 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:39.928241 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:39.928973 31325 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:21:39.929204 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45847us
I20240426 13:21:39.929435 31325 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.930075 25030 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:39.929997 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.930310 25099 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:39.929913 31325 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:39.933271 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:39.888163 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 869) took 2905 ms. Trace:
I20240426 13:21:39.931809 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:39.882418 31299 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.891523 25027 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:39.892304 31339 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:39.951503 31339 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:39.952980 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81620.4us
I20240426 13:21:39.956434 31325 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:39.895499 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2774) took 1989 ms. Trace:
I20240426 13:21:39.960378 31321 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.900492 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1141) took 1711 ms. Trace:
I20240426 13:21:39.960991 24947 rpcz_store.cc:276] 0426 13:21:38.189888 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.789508 (+1599620us) service_pool.cc:225] Handling call
0426 13:21:39.789859 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.790250 (+   391us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:39.790624 (+   374us) log.cc:842] Serialized 14 byte log entry
0426 13:21:39.900143 (+109519us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.900201 (+    58us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:39.900250 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.900322 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.900436 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31251,"prepare.queue_time_us":1024,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":53234,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"thread_start_us":57728,"threads_started":1,"wal-append.queue_time_us":71668}
I20240426 13:21:39.900620 30614 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:39.869750 31287 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:39.963575 31287 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:39.963663 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2775) took 2040 ms. Trace:
I20240426 13:21:39.892380 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:39.901672 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47960.3us
I20240426 13:21:39.927423 31195 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:39.886859 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.964979 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3042) took 1745 ms. Trace:
I20240426 13:21:39.965121 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:39.965569 24951 rpcz_store.cc:276] 0426 13:21:38.219545 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.252444 (+ 32899us) service_pool.cc:225] Handling call
0426 13:21:39.886523 (+1634079us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:21:39.964926 (+ 78403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9054976}
I20240426 13:21:39.931059 31090 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:39.966253 31297 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20240426 13:21:39.966825 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74557.5us
I20240426 13:21:39.967233 31297 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:39.931591 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57371.2us
W20240426 13:21:39.967483 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2773) took 2061 ms (client timeout 1735 ms). Trace:
W20240426 13:21:39.967953 25096 rpcz_store.cc:269] 0426 13:21:37.905930 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.915768 (+  9838us) service_pool.cc:225] Handling call
0426 13:21:39.967128 (+2051360us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.967296 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156870400}
I20240426 13:21:39.968811 25099 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:21:39.971312 31299 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:39.971812 25103 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20240426 13:21:39.933874 25023 rpcz_store.cc:276] 0426 13:21:36.982210 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.983555 (+  1345us) service_pool.cc:225] Handling call
0426 13:21:36.983905 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:37.073760 (+ 89855us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.073801 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:37.073846 (+    45us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:21:37.073882 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.887733 (+2813851us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.887803 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.887966 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206205824}
I20240426 13:21:39.972672 31299 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
W20240426 13:21:39.929041 31282 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 3.024s	user 0.009s	sys 0.013s
I20240426 13:21:39.973071 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2359) took 1828 ms. Trace:
I20240426 13:21:39.948786 25025 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20240426 13:21:39.949113 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2768) took 2264 ms. Trace:
I20240426 13:21:39.963966 31287 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:39.959054 25101 rpcz_store.cc:276] 0426 13:21:37.906309 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.906625 (+   316us) service_pool.cc:225] Handling call
0426 13:21:37.906998 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:37.958661 (+ 51663us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:38.123520 (+164859us) log.cc:842] Serialized 14 byte log entry
0426 13:21:38.470116 (+346596us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:38.470169 (+    53us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:38.470215 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.894349 (+1424134us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b1800451120. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:21:39.895017 (+   668us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.895065 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.895293 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24494,"prepare.queue_time_us":558223,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":6134,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":15047,"spinlock_wait_cycles":128359424,"thread_start_us":504611,"threads_started":2,"wal-append.queue_time_us":434707}
I20240426 13:21:39.975816 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3047) took 1448 ms. Trace:
I20240426 13:21:39.976222 24956 rpcz_store.cc:276] 0426 13:21:38.527339 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.482177 (+954838us) service_pool.cc:225] Handling call
0426 13:21:39.975622 (+493445us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54002ca6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.975748 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306248960}
I20240426 13:21:39.977620 25101 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:39.975031 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:39.878382 25623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 1016) took 2706 ms (client timeout 2200 ms). Trace:
I20240426 13:21:39.974440 25102 rpcz_store.cc:276] 0426 13:21:37.684713 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.686238 (+  1525us) service_pool.cc:225] Handling call
0426 13:21:37.686533 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:37.882703 (+196170us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.882748 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:37.883286 (+   538us) log.cc:842] Serialized 26 byte log entry
0426 13:21:38.079261 (+195975us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:38.079301 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.948527 (+1869226us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.948585 (+    58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.948657 (+    72us) raft_consensus.cc:1671] finished
0426 13:21:39.948670 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.948933 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37374,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":17058,"spinlock_wait_cycles":442842368,"thread_start_us":151698,"threads_started":1,"wal-append.queue_time_us":687510,"wal-append.run_cpu_time_us":2021,"wal-append.run_wall_time_us":1192211}
I20240426 13:21:39.888166 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:39.958174 31135 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:39.979019 25623 rpcz_store.cc:269] 0426 13:21:37.172200 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.172342 (+   142us) service_pool.cc:225] Handling call
0426 13:21:39.878131 (+2705789us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54006b26d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.878290 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5058811264}
I20240426 13:21:39.979027 25617 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:39.914831 31394 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.883824 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:39.964110 25105 rpcz_store.cc:276] 0426 13:21:37.923142 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.923317 (+   175us) service_pool.cc:225] Handling call
0426 13:21:37.923658 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:37.941165 (+ 17507us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:38.124667 (+183502us) log.cc:842] Serialized 14 byte log entry
0426 13:21:38.210962 (+ 86295us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:38.211036 (+    74us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:38.211104 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.963309 (+1752205us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:39.963372 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.963515 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20458,"prepare.queue_time_us":548316,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":206026,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":32674,"spinlock_wait_cycles":39765376,"thread_start_us":249621,"threads_started":2,"wal-append.queue_time_us":751539}
W20240426 13:21:39.946259 31329 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.645s	user 0.000s	sys 0.002s
I20240426 13:21:39.973536 25103 rpcz_store.cc:276] 0426 13:21:38.144293 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.144415 (+   122us) service_pool.cc:225] Handling call
0426 13:21:39.971270 (+1826855us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:39.972280 (+  1010us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.972879 (+   599us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54003e21d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:39.973016 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4200358272}
W20240426 13:21:39.990576 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58853us
I20240426 13:21:39.964257 31325 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:39.997591 31090 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:39.997684 25102 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:39.973065 31299 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:40.000733 25097 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:40.001639 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:40.004184 31361 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:40.004091 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.005435 25096 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:39.997486 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2375) took 1391 ms. Trace:
I20240426 13:21:40.008405 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.009119 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.009747 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:39.980818 31195 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:40.010283 31371 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.011425 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.012832 31352 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.015901 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.016547 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.019222 31292 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.022100 25098 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:40.022598 31339 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.025018 25100 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:40.035737 31398 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.038139 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2530) took 1482 ms. Trace:
I20240426 13:21:40.043430 25104 rpcz_store.cc:276] 0426 13:21:38.605739 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.996608 (+1390869us) service_pool.cc:225] Handling call
0426 13:21:39.996947 (+   339us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.997166 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.997188 (+    22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:39.997255 (+    67us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:39.997297 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:39.997353 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:39.997440 (+    87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:40.053534 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:39.993691 31297 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:40.022907 31396 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.061476 24953 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
W20240426 13:21:40.065085 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49368.2us
I20240426 13:21:40.067027 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.068434 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.069265 25102 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20240426 13:21:40.070029 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.071079 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.016943 31247 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.076197 31120 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:21:40.076304 31247 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.076776 31120 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:40.077270 31120 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:21:40.080250 31352 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:40.080693 31352 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.081136 31352 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:40.011452 31323 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.084833 31323 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.055370 31361 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:40.085332 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:40.085605 31361 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:40.085970 25097 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:40.064694 24953 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:21:40.087867 24953 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20240426 13:21:40.088555 24953 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:40.089560 31193 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20240426 13:21:40.079209 31209 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.034559 31090 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.108347 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.112740 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:40.110968 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3031) took 3209 ms. Trace:
I20240426 13:21:40.116742 25028 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:40.114837 24953 rpcz_store.cc:276] 0426 13:21:36.901168 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:36.901699 (+   531us) service_pool.cc:225] Handling call
0426 13:21:36.902026 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.904074 (+3002048us) spinlock_profiling.cc:243] Waited 2.68 s on lock 0x7b5800adf9b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:21:40.089064 (+184990us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:40.110906 (+ 21842us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6155879168}
I20240426 13:21:40.056366 24948 rpcz_store.cc:276] 0426 13:21:38.556156 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.790037 (+1233881us) service_pool.cc:225] Handling call
0426 13:21:39.790509 (+   472us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:39.790860 (+   351us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:39.799252 (+  8392us) log.cc:842] Serialized 14 byte log entry
0426 13:21:40.037213 (+237961us) log.cc:842] Serialized 14 byte log entry
0426 13:21:40.037674 (+   461us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:40.037739 (+    65us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:40.037799 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:40.037881 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:40.038030 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10555,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":17028,"raft.queue_time_us":247,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":161,"thread_start_us":178896,"threads_started":1}
W20240426 13:21:40.119541 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 235811us
I20240426 13:21:40.120678 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.061280 31297 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:40.124178 25024 raft_consensus.cc:2354] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:40.126893 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.128055 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.130865 25623 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:40.123577 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:40.106915 31090 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:40.134284 25620 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:40.136979 31345 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:40.143244 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:40.146616 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 145072us
W20240426 13:21:40.152866 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 133438us
I20240426 13:21:40.155763 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.155962 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.159797 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.161401 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:40.163126 31308 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:40.162286 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:40.164610 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:40.161911 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.179591 30665 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:40.179932 31350 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:40.180671 31350 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.181123 31195 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.183454 31195 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.183845 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2516) took 3035 ms. Trace:
I20240426 13:21:40.180107 30665 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:40.184271 24950 rpcz_store.cc:276] 0426 13:21:37.148664 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.153765 (+  5101us) service_pool.cc:225] Handling call
0426 13:21:37.154198 (+   433us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:37.166073 (+ 11875us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.166111 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:37.222718 (+ 56607us) log.cc:842] Serialized 26 byte log entry
0426 13:21:39.947959 (+2725241us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b1800145a80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:40.035402 (+ 87443us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:40.035453 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:40.035579 (+   126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:40.139581 (+104002us) raft_consensus.cc:1671] finished
0426 13:21:40.139600 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:40.183646 (+ 44046us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":204191,"raft.queue_time_us":14102,"raft.run_cpu_time_us":5729,"raft.run_wall_time_us":54208,"spinlock_wait_cycles":29663104,"thread_start_us":87064,"threads_started":1,"wal-append.queue_time_us":2891064}
W20240426 13:21:40.188248 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2368) took 1966 ms (client timeout 1960 ms). Trace:
W20240426 13:21:40.188666 25098 rpcz_store.cc:269] 0426 13:21:38.221795 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:38.223967 (+  2172us) service_pool.cc:225] Handling call
0426 13:21:40.188047 (+1964080us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:40.188172 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3708644608}
I20240426 13:21:40.113176 31398 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.094237 31299 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.192744 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.185395 30665 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:40.193302 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.198767 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:40.195227 25097 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:40.200246 25025 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:40.202273 25619 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:40.173184 31369 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:40.212370 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:40.214046 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2522) took 2504 ms. Trace:
I20240426 13:21:40.214491 24955 rpcz_store.cc:276] 0426 13:21:37.709367 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:37.710091 (+   724us) service_pool.cc:225] Handling call
0426 13:21:37.775454 (+ 65363us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:37.775827 (+   373us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:37.775865 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:37.776388 (+   523us) log.cc:842] Serialized 26 byte log entry
0426 13:21:38.014612 (+238224us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:38.015660 (+  1048us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:38.015751 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:39.945413 (+1929662us) spinlock_profiling.cc:243] Waited 8.46 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:40.213612 (+268199us) raft_consensus.cc:1671] finished
0426 13:21:40.213636 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:40.213880 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2072,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":24463,"spinlock_wait_cycles":19466624,"thread_start_us":138653,"threads_started":1,"wal-append.queue_time_us":1328009}
I20240426 13:21:40.217023 31209 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.216584 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 10 to 13, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 13 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:40.200599 31247 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20240426 13:21:40.218820 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:40.220824 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45232.5us
I20240426 13:21:40.221880 31211 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:40.222294 31211 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:40.222210 31276 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:40.222786 31211 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:40.223870 31325 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:40.226390 31299 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:40.228613 31292 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.229260 31292 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:40.231873 31297 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.233153 31297 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:40.236711 31409 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20240426 13:21:40.166412 31287 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.239099 31287 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.240842 31287 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:40.241449 31287 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.243348 31287 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.196952 31308 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.281958 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.283368 31276 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:40.283797 31276 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.284219 31276 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:40.281833 31195 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:21:40.296469 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.297675 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:40.317420 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.328258 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:40.330848 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57294.1us
W20240426 13:21:40.330832 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47145.5us
W20240426 13:21:40.330622 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 182778us
I20240426 13:21:40.162775 31361 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.336268 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:40.343196 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61281.1us
I20240426 13:21:40.351140 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:40.351871 31173 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20240426 13:21:40.352905 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:40.353865 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:40.355329 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.358840 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.360538 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.361582 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.355422 31369 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:40.356081 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:40.362534 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.356576 25624 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:40.346606 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:40.357048 31307 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:40.364867 31307 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:40.365284 31307 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:40.365787 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.366400 24956 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:21:40.356074 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:40.367825 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.361953 31271 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:40.368763 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.383121 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:40.384366 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.389709 25031 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:40.386019 24953 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:40.347097 31305 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:21:40.396441 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.397538 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.361881 31167 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:40.394209 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.402555 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.403141 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:40.405987 24948 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:40.386070 31271 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.411250 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:40.416128 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58608.7us
I20240426 13:21:40.420169 31303 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:21:40.420171 31339 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:40.420866 31303 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:40.421315 31303 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:21:40.421509 31339 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:40.421932 31339 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:40.422796 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:40.404057 31167 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:40.425117 31209 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:40.422855 31339 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.426816 31394 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:40.426824 31339 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:40.427755 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:40.428905 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:40.429288 31271 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:40.402712 31312 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20240426 13:21:40.427906 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 71945.5us
I20240426 13:21:40.429433 25029 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:40.460152 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:40.403879 31170 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.003s
I20240426 13:21:40.468266 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.461776 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:40.401005 31305 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:40.390029 31276 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.467257 24955 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:40.486132 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2781) took 1203 ms. Trace:
I20240426 13:21:40.427417 31394 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:40.529453 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:40.531872 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 107837us
I20240426 13:21:40.486752 25099 rpcz_store.cc:276] 0426 13:21:39.282840 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.971745 (+688905us) service_pool.cc:225] Handling call
0426 13:21:39.972141 (+   396us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:39.972644 (+   503us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.972671 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:39.973310 (+   639us) log.cc:842] Serialized 52 byte log entry
0426 13:21:40.124966 (+151656us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:40.368510 (+243544us) log.cc:842] Serialized 14 byte log entry
0426 13:21:40.381668 (+ 13158us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:40.381731 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:40.485835 (+104104us) raft_consensus.cc:1671] finished
0426 13:21:40.485863 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:40.486059 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3597,"raft.queue_time_us":13293,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":6063,"thread_start_us":367917,"threads_started":2,"wal-append.queue_time_us":383716}
I20240426 13:21:40.549161 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:40.555634 31276 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.459718 31312 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:40.574350 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 89317.9us
I20240426 13:21:40.584637 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:40.536402 31394 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:40.591491 31276 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20240426 13:21:40.640754 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 108032us
I20240426 13:21:40.615219 31276 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:40.643693 31276 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:40.568655 31300 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:40.571017 31312 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:40.475198 31170 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:40.682562 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 222525us
I20240426 13:21:40.689258 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.690019 24950 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:21:40.693432 31374 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:21:40.694981 30602 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:40.704313 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:40.695323 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 218325us
I20240426 13:21:40.708781 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:40.695474 30602 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:40.425141 31167 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:40.704802 31414 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20240426 13:21:40.722659 25025 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
W20240426 13:21:40.726922 31352 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.645s	user 0.021s	sys 0.015s
I20240426 13:21:40.727365 31352 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:40.731757 31090 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.558s	user 0.005s	sys 0.000s
I20240426 13:21:40.734652 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:40.735671 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:40.721386 31209 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20240426 13:21:40.741516 31299 consensus_meta.cc:287] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.515s	user 0.023s	sys 0.015s
I20240426 13:21:40.759411 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.760591 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:40.739353 31193 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:40.765046 25105 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
W20240426 13:21:40.744617 25100 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.056s	sys 0.161s
W20240426 13:21:40.773089 25100 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.057s	sys 0.162s
I20240426 13:21:40.741050 31300 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:40.745640 31361 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.751822 31209 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:40.743163 31090 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.771165 31354 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:40.765226 31290 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:40.789345 31354 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:40.792551 31354 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:40.793792 31297 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.560s	user 0.008s	sys 0.000s
I20240426 13:21:40.796154 31288 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:40.797515 31288 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:40.797992 31288 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.798216 31300 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.759148 30602 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.754658 31305 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:40.784674 31120 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.707s	user 0.011s	sys 0.025s
I20240426 13:21:40.799971 24956 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
W20240426 13:21:40.786124 31326 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.889s	user 0.017s	sys 0.027s
I20240426 13:21:40.840948 24949 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
W20240426 13:21:40.822337 31211 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.599s	user 0.021s	sys 0.014s
I20240426 13:21:40.825304 24950 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:40.786659 31209 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:40.799875 31288 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.852664 25029 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:40.824829 31170 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.800778 31352 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.823678 31090 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.867239 31394 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.815629 31350 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:40.858062 30665 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.656s	user 0.017s	sys 0.018s
I20240426 13:21:40.819219 31398 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.794999 31271 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.790818 31193 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.850564 31120 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.852540 31326 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.878280 31305 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:40.855629 31211 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:40.898036 31167 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:40.883102 30602 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:40.900949 24955 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20240426 13:21:40.893862 31307 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.005s
I20240426 13:21:40.903447 25620 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:21:40.869180 31276 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.906142 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:40.874969 30665 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.905933 31307 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.958269 31307 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:40.960706 31410 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:40.965358 24951 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:40.965857 31303 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.023s
I20240426 13:21:40.966279 31303 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.969229 31193 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:40.969774 31415 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.566s	user 0.001s	sys 0.002s
I20240426 13:21:40.970793 31271 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.971450 30665 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.904825 31312 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:40.971892 25615 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:40.969218 31120 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.973213 31312 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:40.974190 31276 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.982085 31211 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.984073 31394 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.974232 31303 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:40.994197 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.002753 25619 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:40.972875 31326 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:40.826202 31354 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.039054 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:41.040709 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.042081 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.031579 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:41.039054 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:41.045169 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.046059 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.046821 25027 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
I20240426 13:21:41.048017 25026 raft_consensus.cc:2354] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21.
I20240426 13:21:41.049196 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.049492 25624 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:40.963881 31209 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.060732 31354 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.074326 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.136824 31209 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.150202 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.159229 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.161809 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 1148) took 1154 ms. Trace:
I20240426 13:21:41.162113 24949 rpcz_store.cc:276] 0426 13:21:40.007256 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.057054 (+ 49798us) service_pool.cc:225] Handling call
0426 13:21:41.161691 (+1104637us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b54000bed54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:41.161765 (+    74us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1517824256}
I20240426 13:21:41.163812 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2803) took 1107 ms. Trace:
I20240426 13:21:41.164196 25105 rpcz_store.cc:276] 0426 13:21:40.056682 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.072674 (+ 15992us) service_pool.cc:225] Handling call
0426 13:21:40.072922 (+   248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:40.770143 (+697221us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:41.163546 (+393403us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:41.163744 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1579841792}
I20240426 13:21:41.167364 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 1155) took 1077 ms. Trace:
I20240426 13:21:41.167837 24951 rpcz_store.cc:276] 0426 13:21:40.089523 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.109771 (+ 20248us) service_pool.cc:225] Handling call
0426 13:21:41.167143 (+1057372us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:41.167297 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1923819264}
I20240426 13:21:41.176687 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2534) took 1702 ms. Trace:
I20240426 13:21:41.182111 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.177192 24954 rpcz_store.cc:276] 0426 13:21:39.474750 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:39.866690 (+391940us) service_pool.cc:225] Handling call
0426 13:21:39.976820 (+110130us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:39.977491 (+   671us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:39.977529 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:39.978021 (+   492us) log.cc:842] Serialized 26 byte log entry
0426 13:21:40.118530 (+140509us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:40.118613 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:40.118734 (+   121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:40.969108 (+850374us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:40.970791 (+  1683us) raft_consensus.cc:1671] finished
0426 13:21:40.970809 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:41.176487 (+205678us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002ca6d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:41.176624 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":104,"spinlock_wait_cycles":277741568,"thread_start_us":139963,"threads_started":1,"wal-append.queue_time_us":307351}
I20240426 13:21:41.186866 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:41.151480 30665 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20240426 13:21:41.200284 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.156333 31193 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20240426 13:21:41.205982 25024 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 12.
W20240426 13:21:41.219364 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.013s	sys 0.061s
W20240426 13:21:41.219805 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.013s	sys 0.061s
I20240426 13:21:41.186569 31398 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
W20240426 13:21:41.222568 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 839841us
I20240426 13:21:41.249768 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.258042 25620 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:41.267503 31299 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:41.270938 31308 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:41.279632 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.010s	sys 0.038s
W20240426 13:21:41.282476 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.010s	sys 0.038s
W20240426 13:21:41.293279 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.030s	sys 0.025s
W20240426 13:21:41.295399 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.031s	sys 0.026s
W20240426 13:21:41.297837 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.016s	sys 0.061s
W20240426 13:21:41.298470 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.017s	sys 0.062s
I20240426 13:21:41.297118 25023 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
W20240426 13:21:41.301249 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 716696us
W20240426 13:21:41.302511 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.021s	sys 0.041s
W20240426 13:21:41.303788 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.021s	sys 0.041s
W20240426 13:21:41.304492 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 545616us
I20240426 13:21:41.305264 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:41.299209 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.016s	sys 0.016s
I20240426 13:21:41.304631 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:41.311020 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.017s	sys 0.016s
I20240426 13:21:41.305572 31308 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20240426 13:21:41.310350 25619 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:41.306046 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:41.315464 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 602366us
I20240426 13:21:41.315961 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:41.314272 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2406) took 1065 ms. Trace:
I20240426 13:21:41.325050 25619 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:21:41.330578 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:41.324288 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.048s	sys 0.041s
W20240426 13:21:41.332831 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.048s	sys 0.041s
W20240426 13:21:41.333112 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 554919us
W20240426 13:21:41.321404 31300 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.028s	sys 0.040s
I20240426 13:21:41.334436 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.334654 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.317728 31209 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:41.338359 25097 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:41.318673 25028 rpcz_store.cc:276] 0426 13:21:40.244372 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.249184 (+  4812us) service_pool.cc:225] Handling call
0426 13:21:40.302872 (+ 53688us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:40.333697 (+ 30825us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:21:40.381785 (+ 48088us) log.cc:842] Serialized 14 byte log entry
0426 13:21:41.303956 (+922171us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:41.304009 (+    53us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:41.304441 (+   432us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:41.304519 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:41.310109 (+  5590us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34467,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":3974,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3749,"thread_start_us":887911,"threads_started":1}
I20240426 13:21:41.349635 31195 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:41.340210 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.350679 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.351123 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.321702 31287 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:41.352413 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:41.324504 31325 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:41.355895 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51390us
I20240426 13:21:41.353963 25588 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:41.337512 31300 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.029s	sys 0.041s
I20240426 13:21:41.350226 31195 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:41.362493 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56521.7us
I20240426 13:21:41.363430 24948 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
W20240426 13:21:41.354876 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.031s	sys 0.071s
I20240426 13:21:41.358314 24852 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:41.360344 25104 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:41.360818 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.354405 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.365502 25098 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:41.364281 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.366104 25101 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:41.364812 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.031s	sys 0.071s
I20240426 13:21:41.340284 31209 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:41.370599 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.371466 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.371466 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.372792 25102 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:41.358289 31374 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:41.374709 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:41.375370 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 640768us
I20240426 13:21:41.377071 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.377321 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.378057 24956 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:41.341576 31398 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20240426 13:21:41.360399 31326 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:41.379483 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.383074 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.386909 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.365144 31287 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:41.389647 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.372073 31209 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:41.366159 31303 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20240426 13:21:41.374528 31271 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:21:41.392257 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.392747 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.393015 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:41.393872 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.002s	sys 0.031s
I20240426 13:21:41.395932 24951 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:41.396255 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.002s	sys 0.031s
I20240426 13:21:41.371546 31167 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20240426 13:21:41.398211 31308 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.398455 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:41.394666 25100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.016s	sys 0.036s
W20240426 13:21:41.403245 25100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.016s	sys 0.036s
I20240426 13:21:41.402108 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.404575 24956 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:21:41.388389 30602 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:41.407814 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.388234 31374 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:41.410984 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.389199 31326 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.407805 31300 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.005s
I20240426 13:21:41.414819 25617 raft_consensus.cc:2354] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:21:41.409318 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.406296 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.416457 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2796) took 1414 ms. Trace:
I20240426 13:21:41.411298 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.419037 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.421603 25103 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:41.419862 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.420825 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.421855 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.423858 25615 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:41.422137 24947 raft_consensus.cc:2354] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21.
I20240426 13:21:41.424470 25097 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:41.424760 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:41.407395 31325 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:41.427541 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.427747 25029 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:41.431929 25029 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:41.428656 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.435302 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.433755 25026 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:41.435947 25028 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20240426 13:21:41.438586 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.438664 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.409014 31271 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:41.412726 31170 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:41.417486 31326 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:21:41.423698 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1444549040000993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
I20240426 13:21:41.444155 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.445664 31227 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:41.446350 31227 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.450199 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.451052 25097 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:41.452267 25069 leader_election.cc:398] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:21:41.452564 25069 leader_election.cc:401] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:41.455387 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.454313 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.456935 25103 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:41.435179 31211 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:41.453519 25029 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.463291 25104 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:21:41.464422 25099 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
W20240426 13:21:41.449841 31305 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.548s	user 0.007s	sys 0.004s
I20240426 13:21:41.467221 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.468022 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.455554 30665 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
W20240426 13:21:41.469051 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49265.2us
I20240426 13:21:41.423416 31308 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.470697 25029 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:41.471918 25097 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:41.470073 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.473897 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.472704 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.417301 31195 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.477998 25069 leader_election.cc:398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21.
I20240426 13:21:41.478391 25069 leader_election.cc:401] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20240426 13:21:41.460548 31211 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.465654 31325 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:41.481252 25103 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:21:41.472748 30665 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:41.428202 31374 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.421384 25100 rpcz_store.cc:276] 0426 13:21:40.002439 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.059797 (+ 57358us) service_pool.cc:225] Handling call
0426 13:21:40.123220 (+ 63423us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:40.186814 (+ 63594us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:40.774632 (+587818us) log.cc:842] Serialized 14 byte log entry
0426 13:21:41.416221 (+641589us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:41.416275 (+    54us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:41.416306 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:41.416348 (+    42us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:41.416419 (+    71us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":618331,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":14857,"thread_start_us":1142827,"threads_started":2}
I20240426 13:21:41.494258 31350 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:41.475059 31422 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:41.497579 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.497355 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.500825 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.499790 31350 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:41.504666 24954 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:41.505693 24947 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:41.484952 31211 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:41.542402 25620 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:41.546119 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.546731 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:41.547959 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:41.550503 25098 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:41.553999 25102 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:41.502235 30665 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.560690 25097 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:41.558429 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:41.556092 31394 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:41.574772 31135 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:41.556874 31422 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:41.475541 31287 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.586982 25097 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:41.561069 31319 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:41.558959 31428 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:41.566669 31427 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.559638 31167 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20240426 13:21:41.604918 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.600670 31101 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:41.603041 31350 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.610085 31350 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:41.589267 31227 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.606462 25103 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:41.590786 31319 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:41.552395 31271 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.594383 31426 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:41.614377 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.618494 24951 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:41.619807 24952 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20240426 13:21:41.623258 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.623816 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.607349 31425 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:41.603873 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:41.615774 31426 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:41.628177 25100 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
W20240426 13:21:41.627882 31361 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.031s	sys 0.085s
I20240426 13:21:41.640915 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.607316 31428 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.644809 25619 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:41.626646 31425 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:41.646171 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.628882 31325 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:21:41.605374 31427 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:41.640185 31361 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.032s	sys 0.085s
I20240426 13:21:41.636058 31426 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:21:41.667655 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 280784us
I20240426 13:21:41.667865 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.675465 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.646116 31308 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:41.676910 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.648857 31425 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:41.691164 25102 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:41.690821 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 114893us
I20240426 13:21:41.680560 31325 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
W20240426 13:21:41.705996 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 325789us
I20240426 13:21:41.711843 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:41.712859 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 321493us
I20240426 13:21:41.717530 31345 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.717635 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.717929 31170 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:41.718287 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.718400 31170 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.718083 31345 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.720389 25029 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:41.720916 31170 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.721050 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.696455 31435 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:41.697398 31308 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:41.724491 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.725205 25103 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:41.728381 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.649487 31319 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.737579 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:41.728968 31428 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:41.734774 25104 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:41.741184 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:41.741745 25029 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:41.728768 31427 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:41.738091 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.739321 25023 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:41.739806 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:41.744942 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.740326 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:41.743691 25098 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:41.746078 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.746454 25098 raft_consensus.cc:2362] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15. Re-sending same reply.
I20240426 13:21:41.714850 31325 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:41.721438 31170 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:41.749637 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.749763 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:41.721414 31345 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:41.752024 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:41.724826 31167 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20240426 13:21:41.753212 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.753827 31436 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:41.754015 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.755167 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.756156 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:41.757287 25104 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:41.757313 31326 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.758548 25104 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:41.759635 25104 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:41.760772 25104 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:41.747582 31308 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.767897 31308 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:41.768256 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.769577 31247 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:41.770998 31247 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:21:41.771421 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:41.772192 31299 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:41.772540 31299 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:41.773319 31299 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.714036 31303 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20240426 13:21:41.747400 31422 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21.
I20240426 13:21:41.783824 25105 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:41.784286 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2556) took 1401 ms. Trace:
I20240426 13:21:41.784616 24948 rpcz_store.cc:276] 0426 13:21:40.382442 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:40.413854 (+ 31412us) service_pool.cc:225] Handling call
0426 13:21:40.808719 (+394865us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:21:40.808886 (+   167us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:41.369555 (+560669us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:41.784105 (+414550us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:41.784243 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1743312512}
I20240426 13:21:41.787395 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:41.787943 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.780825 31319 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:41.782541 31299 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:41.796864 25098 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20240426 13:21:41.800251 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75842.1us
I20240426 13:21:41.801997 25031 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:21:41.807157 25097 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:41.808864 30665 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:21:41.789196 31439 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:41.787715 31398 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:41.813968 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.814498 25096 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:41.817535 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.802443 31428 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:41.814559 31305 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20240426 13:21:41.803977 31287 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:41.807744 31303 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:41.822109 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 20, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:21:41.824882 31433 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20240426 13:21:41.825486 31436 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:41.825947 31436 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:41.826773 31436 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.820792 31428 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.829432 31326 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:41.829885 31428 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.854633 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:41.818190 31398 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:41.870836 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 312509us
I20240426 13:21:41.856328 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.821723 31167 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:41.797437 31211 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.874219 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 17, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 17 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:41.877012 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:41.877197 31446 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:41.877142 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:41.811548 31209 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.878741 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:41.879742 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51496us
I20240426 13:21:41.891340 31195 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:21:41.822669 31287 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:41.892901 31287 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:41.894667 31425 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:41.896327 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82436.4us
I20240426 13:21:41.896610 25615 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:41.897089 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.833937 31436 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:41.887907 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.809687 31325 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.889401 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.857530 31300 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:41.904371 31300 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.871407 31439 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:41.904909 31195 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:41.887993 31170 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:41.909161 31170 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:41.910233 31170 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.913340 31211 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:41.914333 25103 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:41.915643 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43518.2us
I20240426 13:21:41.918270 31247 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:41.920717 31442 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20240426 13:21:41.923178 31292 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:41.923600 31292 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:41.924387 31292 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:41.890041 31446 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.929567 31446 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:41.930773 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 290) took 37415 ms (client timeout 15000 ms). Trace:
I20240426 13:21:41.932157 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.932372 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 17, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 17 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:41.933387 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.933880 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:41.934027 24852 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 18, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 18 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:41.935113 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:41.936328 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.937666 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.939067 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.924440 31167 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:41.940410 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.935640 24852 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 16, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 16 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:41.939805 24956 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:41.942193 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:41.943204 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:41.904970 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48747.3us
I20240426 13:21:41.944726 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:41.952778 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53368.4us
I20240426 13:21:41.954092 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.888885 31396 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:41.954115 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:41.958079 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:41.955655 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77565.1us
I20240426 13:21:41.947657 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:41.956962 25619 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:41.957080 24955 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:41.907475 31426 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:41.958160 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:41.964176 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.927975 31300 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:41.967437 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:41.967048 24852 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 16 to 18, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 18 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:41.944275 31326 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:41.954793 31211 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:41.955516 31305 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:41.975260 31374 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:41.993906 25101 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:41.954255 31439 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20240426 13:21:41.960467 31195 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
W20240426 13:21:42.098116 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 124247us
I20240426 13:21:42.043435 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:42.080296 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106819us
W20240426 13:21:42.104944 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 121519us
I20240426 13:21:41.963025 31446 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:42.107147 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.110910 24852 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 16, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 16 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:41.966209 31290 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:21:41.976796 31435 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:41.993813 31308 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:42.062350 31227 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:41.976900 31211 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:42.160307 31211 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:42.160715 31148 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:42.162369 31429 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:42.112607 31425 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:42.109165 31170 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:42.165966 31170 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:42.131920 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133967us
I20240426 13:21:42.133301 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:41.973418 31422 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.141s
I20240426 13:21:41.973594 31303 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.141s
I20240426 13:21:41.983668 31300 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:41.977176 31297 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.144s
I20240426 13:21:42.106560 31292 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:21:42.169441 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62335.9us
I20240426 13:21:42.171531 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.170466 31394 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:42.177176 31209 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.173592 31439 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240426 13:21:42.178716 31287 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.179822 24852 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 8 to 9, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 9 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:42.180984 31170 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.185420 31247 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:21:41.931180 24853 rpcz_store.cc:269] 0426 13:21:04.515199 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:04.546579 (+ 31380us) service_pool.cc:225] Handling call
0426 13:21:41.930558 (+37383979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:33.573124 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:33.573590 (+   466us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:33.573620 (+    30us) write_op.cc:427] Acquired schema lock
0426 13:21:33.573635 (+    15us) tablet.cc:660] Decoding operations
0426 13:21:33.574988 (+  1353us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:33.575042 (+    54us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:33.575067 (+    25us) tablet.cc:683] Acquiring locks for 19 operations
0426 13:21:33.690464 (+115397us) tablet.cc:699] Row locks acquired
0426 13:21:33.690486 (+    22us) write_op.cc:260] PREPARE: finished
0426 13:21:33.690622 (+   136us) write_op.cc:270] Start()
0426 13:21:33.690745 (+   123us) write_op.cc:275] Timestamp: P: 1714137693690575 usec, L: 0
0426 13:21:33.690763 (+    18us) op_driver.cc:331] REPLICATION: starting
0426 13:21:33.691364 (+   601us) log.cc:842] Serialized 7450 byte log entry
0426 13:21:34.396413 (+705049us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b540011f420. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a71a8 00007fda902a6dea 00007fda90312e65 00007fda90313309 00007fda93d2804e 00007fda93d27828 00007fda93d2c111 00007fda93d2c0c9 00007fda93d2c059 00007fda93d2c021
0426 13:21:34.638498 (+242085us) op_driver.cc:444] REPLICATION: finished
0426 13:21:34.851547 (+213049us) write_op.cc:300] APPLY: starting
0426 13:21:34.851647 (+   100us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:35.163208 (+311561us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:35.163247 (+    39us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:35.584014 (+420767us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:35.668209 (+ 84195us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0426 13:21:35.668262 (+    53us) write_op.cc:311] APPLY: finished
0426 13:21:35.668632 (+   370us) log.cc:842] Serialized 172 byte log entry
0426 13:21:35.668795 (+   163us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:35.668975 (+   180us) write_op.cc:446] Released schema lock
0426 13:21:35.669036 (+    61us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:35.669350 (+   314us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:35.669384 (+    34us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1015831,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":191563,"apply.run_cpu_time_us":26484,"apply.run_wall_time_us":839549,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83585,"num_ops":19,"prepare.queue_time_us":230408,"prepare.run_cpu_time_us":19429,"prepare.run_wall_time_us":823550,"raft.queue_time_us":793931,"raft.run_cpu_time_us":12969,"raft.run_wall_time_us":183341,"replication_time_us":947639,"spinlock_wait_cycles":50503936,"thread_start_us":1354854,"threads_started":4,"wal-append.queue_time_us":122674,"wal-append.run_cpu_time_us":91876,"wal-append.run_wall_time_us":2893169}]]}
W20240426 13:21:42.190182 31451 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20240426 13:21:42.191143 31449 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:21:42.192314 31303 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:42.261698 31453 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.070s	user 0.000s	sys 0.008s
I20240426 13:21:42.195075 31426 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:42.170025 31361 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:42.040920 31398 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.224814 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:42.252624 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46371.4us
W20240426 13:21:42.253140 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 143507us
W20240426 13:21:42.268389 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 288813us
I20240426 13:21:42.253227 31300 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:42.268816 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:42.269071 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:42.253703 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 293667us
W20240426 13:21:42.261494 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 154172us
W20240426 13:21:42.265218 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93764.3us
I20240426 13:21:42.192926 31195 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:21:42.263470 31422 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:42.270386 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:42.261898 31308 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:42.276219 31459 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.093s	user 0.000s	sys 0.008s
I20240426 13:21:42.275440 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 12 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 19 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:42.276427 31398 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:42.277109 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.277335 31429 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:42.271052 31195 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.269457 31361 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:42.279693 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54982.7us
I20240426 13:21:42.284246 25100 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:42.284677 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:42.264808 31435 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.287364 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.288513 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:42.285651 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.290496 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.292788 31300 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.310322 30665 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:42.310750 30665 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:42.311220 30665 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:42.312624 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.300590 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 14 to 17, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 17 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:21:42.315073 25101 raft_consensus.cc:2354] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:42.315136 31460 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:42.308271 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.365690 25097 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:42.287256 31471 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:42.341573 25104 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20240426 13:21:42.301216 25616 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:21:42.297641 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.315165 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:42.380533 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 98439us
I20240426 13:21:42.382789 31474 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:42.312674 31303 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:42.386018 31303 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:42.386376 31303 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:42.388422 31422 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:42.296587 31326 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:42.293471 31435 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.368393 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.380558 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:42.390460 25101 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:42.315958 31425 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
W20240426 13:21:42.382582 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 74387.9us
I20240426 13:21:42.374472 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:42.391166 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84378.2us
I20240426 13:21:42.291241 31211 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.408079 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.410773 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.295737 31101 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:42.393402 25096 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:42.289034 31308 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.414129 24955 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:42.399448 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:42.397503 31271 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.024s	sys 0.005s
I20240426 13:21:42.408569 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:42.400091 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102516us
I20240426 13:21:42.399354 24947 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:42.378751 31427 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:42.414131 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.382084 31471 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:42.415719 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50484.6us
I20240426 13:21:42.415959 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:42.400547 31170 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:42.367513 31374 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:42.299085 31287 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.294309 31396 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:21:42.404426 30602 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:42.420782 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:42.368649 31292 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
I20240426 13:21:42.434505 25028 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20240426 13:21:42.420939 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:42.414667 31101 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:42.292203 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.393985 31425 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:42.423607 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.445585 25026 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:42.425477 24948 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:42.445880 31425 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.425781 31374 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.424611 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.426220 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.448779 31374 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.429669 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.450634 25031 raft_consensus.cc:2362] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21. Re-sending same reply.
I20240426 13:21:42.419557 31326 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:42.452996 24955 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20240426 13:21:42.453632 31326 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:42.454129 31326 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:42.454550 31101 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:42.416148 31271 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.024s	sys 0.005s
I20240426 13:21:42.290171 31361 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.438269 31247 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:21:42.438645 31312 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:42.420024 31427 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.411648 31462 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:21:42.421190 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.444389 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:42.436843 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:42.442474 30665 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.447243 31425 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:42.292372 31339 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.124s
I20240426 13:21:42.468973 25623 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20240426 13:21:42.455940 31101 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:42.473347 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:42.437685 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 145537us
I20240426 13:21:42.507083 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.507627 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.466600 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:42.506394 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 85278.1us
I20240426 13:21:42.426575 31345 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:42.290995 31195 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.433805 30602 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.469419 31439 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:42.461830 31211 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:42.502543 31312 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:42.531008 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64516.7us
I20240426 13:21:42.531630 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.532508 31469 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:42.532675 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.534240 31374 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:42.535113 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.535887 31398 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:42.536170 31469 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:42.527387 25024 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:42.538293 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:42.538561 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:42.541682 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68432.2us
I20240426 13:21:42.546398 25103 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:42.547529 25098 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:42.515817 31422 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:42.520545 31287 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:42.552848 25031 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:42.520745 31396 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:21:42.539978 31312 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:42.516074 31446 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20240426 13:21:42.519637 30602 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.592875 30602 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:42.596429 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67837.4us
I20240426 13:21:42.596616 31209 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:42.547422 31305 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:42.597222 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.597678 31305 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.598637 31422 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:21:42.598883 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.599635 31305 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:42.600652 31361 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:42.579769 25027 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:42.579602 31463 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.064s
I20240426 13:21:42.581259 31444 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:42.581954 31476 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20240426 13:21:42.583003 25097 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:42.541749 31339 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:42.587939 31326 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.520483 31345 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.529253 31439 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.598981 31422 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:42.554715 31427 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.579222 25100 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:42.577675 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:42.608589 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.617205 24955 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:42.553301 31435 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:42.612517 31211 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:42.612465 31469 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:42.661218 25105 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20240426 13:21:42.652767 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75174.7us
I20240426 13:21:42.642791 31271 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:42.664746 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.657224 31305 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:42.652663 31422 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:21:42.669924 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 71094.7us
I20240426 13:21:42.670647 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.660282 31247 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:42.642416 31339 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.674490 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.675740 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.676764 25070 leader_election.cc:398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21.
I20240426 13:21:42.712209 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.648813 31303 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.666719 31271 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:42.717819 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:42.670084 31398 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:21:42.725239 31484 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:21:42.725379 31439 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.717103 31394 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:42.668195 31345 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:42.726862 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51240.8us
I20240426 13:21:42.729012 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.731431 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.675330 31427 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:42.714462 31305 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20240426 13:21:42.722292 31312 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.733672 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:42.735013 31489 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240426 13:21:42.735930 31271 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.675593 30665 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.715651 31396 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:42.741551 24921 leader_election.cc:398] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:42.744066 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.745703 24921 leader_election.cc:401] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:42.675841 31247 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:42.733752 25105 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:42.676959 31469 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:42.748827 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:42.757319 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.758491 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.742717 31463 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:42.761305 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:42.760012 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.737555 31135 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20240426 13:21:42.763475 30665 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:42.750715 31247 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:42.776935 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:42.752321 31439 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:42.780748 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.761917 31211 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:42.784409 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.769824 31488 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:42.766470 31290 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:42.795156 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:42.776237 31476 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:42.750818 31396 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.766256 31436 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20240426 13:21:42.808429 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:42.809437 25100 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:42.799829 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:42.803954 24948 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:42.811597 25027 pending_rounds.cc:76] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 15
I20240426 13:21:42.811986 25027 pending_rounds.cc:97] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 17.16
I20240426 13:21:42.812296 25027 raft_consensus.cc:2866] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:42.812362 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:42.793159 31271 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:42.821743 25024 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:42.814752 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.815479 31339 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.782001 31300 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20240426 13:21:42.819473 31227 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:42.813871 25105 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:42.817739 31490 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:42.783386 31209 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20240426 13:21:42.793967 31369 raft_consensus.cc:2741] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20240426 13:21:42.795634 31326 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:42.826356 31495 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:42.827797 31476 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:42.830250 25030 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:42.795871 31461 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:42.830885 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:42.835752 31490 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:42.909870 31490 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:42.911281 31490 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:42.837213 31227 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.845404 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.913584 31227 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:42.846735 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:42.914064 31429 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:42.848090 31101 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:42.914942 31101 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.914783 31429 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.848323 31312 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:42.848860 31303 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:42.662482 31292 consensus_queue.cc:1159] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 18 status { last_received { term: 17 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:21:42.853204 31396 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:42.854825 31308 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:42.858151 31195 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:42.919447 25623 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20240426 13:21:42.837039 31469 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:42.861992 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47319.3us
W20240426 13:21:42.923683 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 92883.6us
I20240426 13:21:42.903014 25031 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:42.795964 31299 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20240426 13:21:42.920545 24954 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:42.914793 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:42.928050 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:42.930147 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:42.930743 25615 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:42.921389 31435 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:42.935591 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:42.922962 31374 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:42.925804 31495 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:42.929992 31303 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:42.944479 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:42.948168 31350 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:42.948642 24953 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20240426 13:21:43.009181 24953 pending_rounds.cc:76] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:21:43.009492 24953 pending_rounds.cc:97] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20240426 13:21:43.009790 24953 raft_consensus.cc:2866] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:42.985368 31504 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20240426 13:21:42.946291 31374 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:43.015614 25096 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20240426 13:21:43.016414 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61416.9us
I20240426 13:21:42.931061 31461 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20240426 13:21:42.993259 31300 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.000646 31101 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:42.941291 31474 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:42.946979 31476 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:43.002741 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:43.003132 31247 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:42.933560 31319 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:43.005506 25623 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:43.006285 31495 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:42.991484 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68630.3us
I20240426 13:21:43.025970 25615 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:43.026401 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82005.5us
I20240426 13:21:43.034231 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:43.034785 25620 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:21:43.024310 31463 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:43.036489 24953 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:43.020074 31350 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.036504 24947 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:43.037848 24955 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:43.037365 31247 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:43.038637 31350 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:43.024641 31474 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:43.039999 31287 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:43.043368 31287 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:43.024394 31300 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.047255 31374 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:42.995890 31422 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:43.060729 31247 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:43.061161 31247 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.062212 31422 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:43.061903 31247 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.062089 31471 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.063135 31247 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:43.064023 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:43.067689 31300 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:43.035560 31361 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20240426 13:21:43.042455 31514 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:43.069389 31514 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.069451 31476 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:43.070277 31514 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.070528 31476 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.071255 31514 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:43.041525 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:43.037102 31432 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:43.029757 31509 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:43.031935 31319 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.048916 31474 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:43.088526 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.089444 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.100330 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:43.101042 24955 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:43.101330 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.101765 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.102125 25027 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20240426 13:21:43.102829 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.103520 25027 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:43.096818 31195 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.104265 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:43.104775 31455 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.114s	user 0.005s	sys 0.000s
I20240426 13:21:43.104440 31195 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.105095 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.099959 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.106199 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.106873 25025 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:43.107116 31195 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:43.107537 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.098399 31299 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:43.109095 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.094839 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.110807 25105 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:43.112335 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.112818 25028 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:43.113662 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.114380 25028 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20240426 13:21:43.115275 25028 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:43.102839 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.126132 25031 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
W20240426 13:21:43.097513 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 356834us
I20240426 13:21:43.131027 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20240426 13:21:43.133236 25024 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:43.130928 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.141505 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.145188 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.139834 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:43.139822 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 98380.8us
I20240426 13:21:43.147967 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.150496 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:43.151311 25030 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:43.096390 31509 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.153228 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:43.153492 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.087587 31432 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.096560 31422 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:21:43.160331 25096 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 22 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20240426 13:21:43.161286 31509 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:43.094769 31462 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20240426 13:21:43.130192 31435 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.161638 31297 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:21:43.154045 31299 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:21:43.162672 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.155315 31339 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
W20240426 13:21:43.165736 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 253707us
I20240426 13:21:43.154896 31148 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:43.162201 31290 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.015s
I20240426 13:21:43.172287 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.172497 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.174373 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.174691 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.188724 25022 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:43.152315 31312 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:43.181907 25027 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:43.191222 25096 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:43.165426 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.167407 31271 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:43.162617 31394 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:43.191275 25100 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:43.202210 31519 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20240426 13:21:43.181478 31297 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.200935 31148 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:43.207437 31297 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:43.201668 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:43.207970 31148 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.194845 25023 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:43.183431 31492 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:43.200601 31503 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20240426 13:21:43.184124 31299 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.202101 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.181285 31435 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:43.205940 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.211273 25104 raft_consensus.cc:595] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.225539 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:43.202621 31517 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20240426 13:21:43.225255 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.225641 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.226327 25025 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:43.229535 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:43.231174 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:43.234891 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:43.211448 31394 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.234454 31492 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:43.230484 31429 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:43.245512 31292 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.245247 31492 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.245944 31292 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.247766 31292 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:43.249877 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.250469 25104 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:43.251698 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:43.252446 31518 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20240426 13:21:43.252650 31292 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:43.253041 31292 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.253654 31292 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:43.256476 31319 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:43.260092 31394 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:43.260298 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:43.232623 31471 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:43.261934 31292 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.264111 31247 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:21:43.264478 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:43.264988 25104 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:43.265719 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:43.266242 25615 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:43.269866 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:43.270936 25104 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:43.270462 24955 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 11.
I20240426 13:21:43.280926 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.290304 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:43.309679 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 125156us
I20240426 13:21:43.309813 31339 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:43.311173 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.310873 31339 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.221896 31299 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:21:43.272241 31300 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:43.321242 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:43.326339 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.327414 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:43.329411 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:43.307053 31312 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Stepping down as leader of term 17
W20240426 13:21:43.333262 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 108315us
I20240426 13:21:43.306849 31287 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:43.335268 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:43.319470 31462 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:43.336720 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.340484 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:43.319963 31300 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:43.379837 31300 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:43.220911 31432 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.382290 31432 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:43.382812 31271 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:43.325488 31101 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:43.383638 31101 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.354470 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:43.354950 31398 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:43.384718 31398 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.384546 31101 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.385632 31398 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.333715 31312 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:43.387804 31247 consensus_queue.cc:1159] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1590f80bdade41c6a755b8501c55fe16" responder_term: 13 status { last_received { term: 13 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 13 index: 11 } }
I20240426 13:21:43.388473 31495 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:43.334728 31436 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:43.389070 31495 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:43.315320 31297 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:43.389600 31495 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:43.390880 31297 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:21:43.333045 31422 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:43.392217 31462 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:43.391958 31422 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.392494 25023 raft_consensus.cc:2354] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
I20240426 13:21:43.393071 31462 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.396561 25031 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:21:43.355094 31497 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20240426 13:21:43.355927 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 130367us
I20240426 13:21:43.406461 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.356470 31461 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:43.357394 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:43.412847 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55524.2us
I20240426 13:21:43.413928 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.357595 31148 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:43.358134 25100 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:43.342171 31292 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:43.359624 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.360370 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.361984 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.375180 31429 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:43.319654 31396 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20240426 13:21:43.414422 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.416697 25104 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:21:43.428558 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:43.435232 24953 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:43.436296 25024 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:43.423110 25025 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:21:43.446632 25026 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:43.447687 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.439504 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:43.403623 31325 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.463559 25029 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20240426 13:21:43.424156 31369 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:43.453346 25027 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:43.448419 25098 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:43.417557 31308 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240426 13:21:43.443770 31497 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:43.467010 25102 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
I20240426 13:21:43.438849 31429 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:43.477350 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:43.449901 31461 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:43.554693 25096 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:43.555660 31510 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.103s	user 0.007s	sys 0.000s
I20240426 13:21:43.410106 31435 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.569380 25098 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
W20240426 13:21:43.570659 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 156313us
I20240426 13:21:43.574738 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:43.575919 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.576197 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.577466 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.581130 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.582315 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.387820 31312 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.587036 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:43.570065 31504 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20240426 13:21:43.588366 31446 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:43.608745 31135 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:43.609308 31446 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:43.609930 31446 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:43.610517 31135 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:43.610891 31446 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.612587 31287 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:43.612958 31287 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:43.613786 31287 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.615152 31287 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:43.617316 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:43.621479 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.475785 31325 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.585506 31326 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:43.585183 31394 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:43.559474 31474 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:43.569917 31519 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.116s	user 0.000s	sys 0.007s
I20240426 13:21:43.591636 31461 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
W20240426 13:21:43.633975 31501 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:21:43.639940 25099 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:43.651461 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:43.631644 24953 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:43.625815 31465 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:43.632613 31247 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:43.678650 25102 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:43.679934 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:43.792647 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 112839us
I20240426 13:21:43.648470 31326 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:43.693850 31504 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20240426 13:21:43.599722 31509 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.647385 31297 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.644763 31209 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.696545 25030 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20240426 13:21:43.766300 31148 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:21:43.769026 31526 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:21:43.779023 31325 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:43.737807 31488 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.700161 31101 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:21:43.789199 24947 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:43.660904 31461 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:43.728554 31521 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20240426 13:21:43.574081 31429 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.827683 31422 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:43.830090 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:43.794677 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:43.797649 31435 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:43.830770 25099 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:43.807173 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:43.831981 25102 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:21:43.832387 31474 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:43.834986 31429 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:43.836392 25031 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:43.839557 31193 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:43.842973 31312 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:43.852092 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:43.852679 31432 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:43.800740 31490 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:21:43.811164 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:43.807289 31514 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.736s	user 0.016s	sys 0.004s
I20240426 13:21:43.823398 31488 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.814934 31461 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.806001 31148 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:43.853838 24953 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:43.856962 25100 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
W20240426 13:21:43.858667 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276424us
I20240426 13:21:43.839658 25099 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:43.920795 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:43.901692 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43945.1us
I20240426 13:21:43.923228 25105 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
W20240426 13:21:43.827978 31299 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.505s	user 0.004s	sys 0.003s
I20240426 13:21:43.804029 25030 pending_rounds.cc:76] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 7
I20240426 13:21:43.920738 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.929020 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:43.929615 25030 pending_rounds.cc:97] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20240426 13:21:43.930115 25030 raft_consensus.cc:2866] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:43.935317 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:43.862435 25102 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 12.
I20240426 13:21:43.937400 25096 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:21:43.939018 31308 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:43.939426 31308 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:43.939754 31308 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:43.941850 31394 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:43.943734 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:43.945860 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.905674 31490 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:43.946848 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:43.807010 31209 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:43.812304 31297 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:43.946955 31394 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:43.935142 31193 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:43.936290 31290 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:43.998245 31398 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.036s	sys 0.060s
I20240426 13:21:44.005353 31303 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:44.005924 31303 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.008267 31303 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:44.005554 31398 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.036s	sys 0.061s
I20240426 13:21:43.948988 31374 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
W20240426 13:21:44.061266 25104 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.007s	sys 0.007s
W20240426 13:21:44.061761 25104 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.007s	sys 0.007s
I20240426 13:21:43.942831 31325 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20240426 13:21:44.003157 31429 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:43.943284 31299 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.003372 31460 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:44.034945 31462 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.004s	sys 0.009s
I20240426 13:21:43.997984 31432 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.077s
I20240426 13:21:44.067540 31195 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.016023 31469 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.128s
I20240426 13:21:44.075294 31374 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:43.953186 31326 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.016608 31290 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.205654 31193 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:44.206151 31193 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:44.137085 31435 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:44.159343 31539 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20240426 13:21:44.147234 31462 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.005s	sys 0.009s
I20240426 13:21:44.149636 31471 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.145577 31543 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:44.150103 31474 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:44.237334 31436 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.154192 31345 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:44.258198 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:44.061563 31148 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.146373 31460 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.298621 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:44.196125 31545 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:44.271816 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 1341) took 1031 ms. Trace:
W20240426 13:21:44.330348 31463 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.015s	sys 0.012s
W20240426 13:21:44.390098 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.013s	sys 0.036s
W20240426 13:21:44.398430 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.013s	sys 0.037s
I20240426 13:21:44.395666 25623 rpcz_store.cc:276] 0426 13:21:43.224569 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.224725 (+   156us) service_pool.cc:225] Handling call
0426 13:21:44.248484 (+1023759us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:44.255651 (+  7167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1796747648}
I20240426 13:21:44.396787 24951 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:44.249264 31509 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:44.344525 31446 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.021s	sys 0.013s
I20240426 13:21:44.160254 31195 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:44.386746 31497 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.827s	user 0.010s	sys 0.000s
I20240426 13:21:44.135803 31429 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.163195 31490 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:44.242112 31435 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:44.397917 31350 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.013s	sys 0.038s
W20240426 13:21:44.398133 31463 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.016s	sys 0.013s
I20240426 13:21:44.391049 31543 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:44.443187 31461 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.019s	sys 0.031s
I20240426 13:21:44.468796 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:44.459530 31369 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.972s	user 0.008s	sys 0.003s
W20240426 13:21:44.470043 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.001s	sys 0.035s
W20240426 13:21:44.563134 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.001s	sys 0.039s
W20240426 13:21:44.563494 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 615661us
I20240426 13:21:44.391721 31488 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.457404 31469 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:44.452247 31497 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:44.462158 31350 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.014s	sys 0.043s
W20240426 13:21:44.464413 31312 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.000s
W20240426 13:21:44.562881 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46522.3us
W20240426 13:21:44.716351 31550 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
W20240426 13:21:44.427899 31300 consensus_meta.cc:287] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.047s	user 0.009s	sys 0.003s
I20240426 13:21:44.717242 31543 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.721179 31435 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.385450 31436 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:44.643990 25101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.015s	sys 0.035s
W20240426 13:21:44.624388 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.030s	sys 0.038s
I20240426 13:21:44.564589 25070 leader_election.cc:398] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
I20240426 13:21:44.393492 31345 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:44.673987 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.014s
I20240426 13:21:44.695101 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:44.743960 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.097s	user 0.011s	sys 0.023s
I20240426 13:21:44.744264 25623 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:21:44.693269 31461 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.027s	sys 0.035s
W20240426 13:21:44.706406 31561 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.256s	user 0.000s	sys 0.000s
I20240426 13:21:44.561641 31135 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.695122 31469 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:44.707160 31560 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.294s	user 0.000s	sys 0.000s
W20240426 13:21:44.712265 31544 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.567s	user 0.001s	sys 0.000s
W20240426 13:21:44.713024 31547 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.459s	user 0.000s	sys 0.000s
I20240426 13:21:44.717532 31300 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.297670 31471 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.730496 31497 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:44.731902 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.044s	sys 0.067s
W20240426 13:21:44.731588 25101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.017s	sys 0.040s
I20240426 13:21:44.508882 31319 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.253s
W20240426 13:21:44.626753 31530 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
W20240426 13:21:44.515875 31476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.017s	sys 0.050s
I20240426 13:21:44.533617 31292 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:44.732348 31495 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.342s	user 0.011s	sys 0.000s
I20240426 13:21:44.565474 31540 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.517127 31209 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:44.560511 31297 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.682782 31299 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.570648 31369 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:44.795858 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.052s	sys 0.035s
W20240426 13:21:44.880703 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.052s	sys 0.035s
I20240426 13:21:44.732165 25070 leader_election.cc:401] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20240426 13:21:44.883196 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 318747us
I20240426 13:21:44.749750 31463 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:44.821097 31551 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
W20240426 13:21:44.821919 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.004s	sys 0.062s
W20240426 13:21:44.895972 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.099s	user 0.005s	sys 0.062s
I20240426 13:21:44.823052 31436 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:44.824891 31509 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.800635 31432 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.837656 24956 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:44.788703 31290 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.916110 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2417) took 1503 ms. Trace:
I20240426 13:21:44.803165 31474 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:44.852459 31195 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.852571 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2452) took 1018 ms. Trace:
I20240426 13:21:44.843369 31495 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.509263 31374 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:44.819304 31422 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.010s	sys 0.032s
W20240426 13:21:44.836087 31476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.017s	sys 0.059s
I20240426 13:21:44.877832 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2608) took 1589 ms. Trace:
W20240426 13:21:44.865659 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.008s
W20240426 13:21:44.872289 31394 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.035s	sys 0.047s
I20240426 13:21:44.901850 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3105) took 1072 ms. Trace:
W20240426 13:21:44.860018 31308 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.920s	user 0.004s	sys 0.004s
I20240426 13:21:44.982795 24947 rpcz_store.cc:276] 0426 13:21:43.829628 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.830282 (+   654us) service_pool.cc:225] Handling call
0426 13:21:43.830570 (+   288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.830914 (+   344us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.830947 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.831400 (+   453us) log.cc:842] Serialized 26 byte log entry
0426 13:21:44.214353 (+382953us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:44.214408 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:44.214482 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:44.901607 (+687125us) raft_consensus.cc:1671] finished
0426 13:21:44.901636 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:44.901789 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7501,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":382540,"threads_started":1,"wal-append.queue_time_us":417316}
I20240426 13:21:44.770273 31135 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:44.865595 31193 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.659s	user 0.009s	sys 0.000s
W20240426 13:21:44.863485 31247 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.066s	user 0.008s	sys 0.000s
I20240426 13:21:44.998776 31247 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:44.942915 31542 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:44.871433 31429 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:44.888192 31460 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:44.930271 25097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.034s	sys 0.026s
W20240426 13:21:45.010267 25097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s	user 0.034s	sys 0.026s
I20240426 13:21:44.814731 31497 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.920893 25103 rpcz_store.cc:276] 0426 13:21:43.412710 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.414826 (+  2116us) service_pool.cc:225] Handling call
0426 13:21:43.415242 (+   416us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.415629 (+   387us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.415669 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.567827 (+152158us) log.cc:842] Serialized 26 byte log entry
0426 13:21:43.788254 (+220427us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:43.788289 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:43.788349 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:44.915855 (+1127506us) raft_consensus.cc:1671] finished
0426 13:21:44.915880 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:44.916044 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2181,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":29574,"thread_start_us":186340,"threads_started":1,"wal-append.queue_time_us":431161}
I20240426 13:21:44.943295 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20240426 13:21:44.943081 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:44.947383 31570 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:44.950150 31568 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:44.951782 31227 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:44.952555 31567 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 11. Current term is 12.
I20240426 13:21:44.955945 31552 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:44.956692 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2427) took 1370 ms. Trace:
I20240426 13:21:44.940443 31474 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.957589 31471 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:44.921018 31509 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:44.921023 31325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.017s	sys 0.049s
I20240426 13:21:44.846582 31292 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.960173 31495 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:44.963469 31345 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.930315 25027 rpcz_store.cc:276] 0426 13:21:43.834470 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.834611 (+   141us) service_pool.cc:225] Handling call
0426 13:21:43.835003 (+   392us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:43.835323 (+   320us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.835350 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:43.835867 (+   517us) log.cc:842] Serialized 52 byte log entry
0426 13:21:44.191295 (+355428us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:44.191336 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:44.191409 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:44.852351 (+660942us) raft_consensus.cc:1671] finished
0426 13:21:44.852381 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:44.852516 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3408,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":6912,"thread_start_us":355059,"threads_started":1,"wal-append.queue_time_us":420973}
W20240426 13:21:44.942445 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.035s	sys 0.034s
W20240426 13:21:44.982609 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.013s	sys 0.009s
I20240426 13:21:44.989696 31308 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.042968 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:44.980957 24955 rpcz_store.cc:276] 0426 13:21:43.288551 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.288658 (+   107us) service_pool.cc:225] Handling call
0426 13:21:43.314940 (+ 26282us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.315252 (+   312us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.315279 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.315647 (+   368us) log.cc:842] Serialized 26 byte log entry
0426 13:21:43.857382 (+541735us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:43.928465 (+ 71083us) log.cc:842] Serialized 14 byte log entry
0426 13:21:43.928660 (+   195us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:43.928723 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:43.939877 (+ 11154us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b180040db00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:44.865999 (+926122us) raft_consensus.cc:1671] finished
0426 13:21:44.877598 (+ 11599us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:44.877768 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55150,"prepare.queue_time_us":77003,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":3827,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":13744,"spinlock_wait_cycles":10714368,"thread_start_us":471514,"threads_started":1,"wal-append.queue_time_us":824986}
I20240426 13:21:45.043530 25030 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:44.986398 25616 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:21:44.984535 31394 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.924s	user 0.036s	sys 0.047s
I20240426 13:21:44.878202 31540 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:44.966451 31422 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.010s	sys 0.033s
I20240426 13:21:44.996731 25622 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:21:44.997478 31135 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:44.997874 31193 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:44.916808 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:44.930951 31290 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:45.010890 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.053s	sys 0.064s
I20240426 13:21:44.924901 31432 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:44.905058 31463 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:45.021595 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.013s	sys 0.004s
W20240426 13:21:45.065657 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.013s	sys 0.004s
W20240426 13:21:45.021836 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.003s	sys 0.015s
W20240426 13:21:45.022590 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.006s	sys 0.013s
W20240426 13:21:45.068226 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.003s	sys 0.015s
W20240426 13:21:45.068287 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.006s	sys 0.013s
I20240426 13:21:45.017153 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:44.957202 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2453) took 1115 ms. Trace:
W20240426 13:21:45.024654 25104 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.008s	sys 0.026s
I20240426 13:21:45.024111 25098 rpcz_store.cc:276] 0426 13:21:43.586215 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.586719 (+   504us) service_pool.cc:225] Handling call
0426 13:21:44.956504 (+1369785us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:44.956635 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2707145600}
I20240426 13:21:45.014747 31541 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.024333 31570 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:45.040755 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.036s	sys 0.035s
I20240426 13:21:45.045032 25029 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:21:45.072587 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.022298 31568 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:45.033111 31325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.017s	sys 0.049s
I20240426 13:21:45.024840 31469 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.010475 31542 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.031901 31227 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.051944 25099 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:21:45.127980 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:45.133492 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:45.052361 25027 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20240426 13:21:45.149466 25023 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:21:45.059437 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.053s	sys 0.065s
I20240426 13:21:45.032490 31572 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:45.032665 31300 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:45.070725 25104 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.008s	sys 0.026s
I20240426 13:21:45.074522 31432 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.076023 31463 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.070106 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:45.076377 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.082152 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2430) took 1226 ms. Trace:
I20240426 13:21:45.127720 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.077593 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.127188 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.078114 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.127646 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:45.169226 24954 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:45.133787 24949 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:45.094907 31308 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.134021 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.172744 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:45.146826 25098 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:45.173331 25620 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:45.173362 25099 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.078869 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.131198 25622 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
W20240426 13:21:45.133730 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61717.2us
I20240426 13:21:45.178012 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:45.096741 31540 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:45.046017 31394 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:45.154415 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.155550 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:45.155822 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.156783 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.156888 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1118) took 1717 ms. Trace:
I20240426 13:21:45.157413 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.157501 31292 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.187923 25619 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:45.158030 31566 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:45.159837 31509 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.160988 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.162181 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3100) took 1421 ms. Trace:
I20240426 13:21:45.162523 31193 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.163362 31497 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.164002 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.165441 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:45.202021 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.163707 31572 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.196763 31436 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.203619 31570 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.203493 31436 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.205034 31568 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.205459 31436 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.196965 31398 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.206259 31398 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.208194 31398 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.199200 31566 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.148523 31567 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20240426 13:21:45.199416 31463 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.211352 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2435) took 1261 ms. Trace:
I20240426 13:21:45.211736 31463 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.211866 25105 rpcz_store.cc:276] 0426 13:21:43.950062 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.950200 (+   138us) service_pool.cc:225] Handling call
0426 13:21:43.950582 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:44.414517 (+463935us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:44.420836 (+  6319us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:44.460792 (+ 39956us) log.cc:842] Serialized 26 byte log entry
0426 13:21:45.015149 (+554357us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:45.015278 (+   129us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:45.016114 (+   836us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:45.211111 (+194997us) raft_consensus.cc:1671] finished
0426 13:21:45.211133 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:45.211294 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":440378,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":18618,"thread_start_us":849929,"threads_started":2,"wal-append.queue_time_us":680772}
I20240426 13:21:45.213057 31350 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.213619 31350 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.213809 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.214640 25105 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:45.215363 31350 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.167881 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:45.078317 25623 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:45.083670 31577 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.216747 25030 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:21:45.179986 31345 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.077827 31135 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.224346 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.078549 31465 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:45.197962 30665 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:21:45.172137 25027 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:21:45.179600 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.180613 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:45.232256 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.167186 24952 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:45.147598 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.176321 31303 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.236650 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.238322 25619 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 21: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:45.188268 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.189173 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.236528 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:45.187866 25104 rpcz_store.cc:276] 0426 13:21:43.440035 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.442332 (+  2297us) service_pool.cc:225] Handling call
0426 13:21:43.442710 (+   378us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.442919 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.442954 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.443426 (+   472us) log.cc:842] Serialized 26 byte log entry
0426 13:21:44.062304 (+618878us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:45.156346 (+1094042us) log.cc:842] Serialized 14 byte log entry
0426 13:21:45.156611 (+   265us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:45.156677 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:45.156710 (+    33us) raft_consensus.cc:1671] finished
0426 13:21:45.156721 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:45.156833 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":961488,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":4594,"raft.queue_time_us":3306,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":1528269,"threads_started":2,"wal-append.queue_time_us":999873}
I20240426 13:21:45.190789 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.063398 31290 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.193464 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.243394 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.243521 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:45.227252 31465 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:45.193953 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.168376 31299 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:45.171553 31469 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.195524 24956 rpcz_store.cc:276] 0426 13:21:43.740778 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.740939 (+   161us) service_pool.cc:225] Handling call
0426 13:21:43.741220 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:44.915457 (+1174237us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:45.161990 (+246533us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:45.162128 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2504073984}
I20240426 13:21:45.195818 25096 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 13: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:45.246769 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.177163 31361 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.249541 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.213898 31429 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20240426 13:21:45.166656 25097 rpcz_store.cc:276] 0426 13:21:43.855615 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.855749 (+   134us) service_pool.cc:225] Handling call
0426 13:21:43.856049 (+   300us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.864434 (+  8385us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:43.864462 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.947148 (+ 82686us) log.cc:842] Serialized 26 byte log entry
0426 13:21:45.010655 (+1063507us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:45.010694 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:45.010758 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:45.081914 (+ 71156us) raft_consensus.cc:1671] finished
0426 13:21:45.081941 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:45.082089 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8165,"raft.queue_time_us":6969,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":7783,"thread_start_us":933378,"threads_started":1,"wal-append.queue_time_us":1022509}
I20240426 13:21:45.186545 31471 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.222602 25023 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:45.264549 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:45.264940 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.265297 25097 raft_consensus.cc:595] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.265789 25097 raft_consensus.cc:603] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:45.222208 25099 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:45.223095 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.186686 31460 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.185249 31474 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.070535 25031 rpcz_store.cc:276] 0426 13:21:43.841694 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:43.841837 (+   143us) service_pool.cc:225] Handling call
0426 13:21:43.842153 (+   316us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:43.842688 (+   535us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:43.842719 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:43.843147 (+   428us) log.cc:842] Serialized 26 byte log entry
0426 13:21:44.802970 (+959823us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:44.803012 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:44.803090 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:44.957024 (+153934us) raft_consensus.cc:1671] finished
0426 13:21:44.957041 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:44.957155 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":73,"thread_start_us":959436,"threads_started":1,"wal-append.queue_time_us":1112317}
I20240426 13:21:45.227305 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.186472 31432 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.228111 24948 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:45.148442 31541 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.228894 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.232883 25030 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:45.272575 25100 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:45.275621 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.187896 24954 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:45.239359 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:45.239928 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.277720 25615 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:21:45.240106 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.213819 31463 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.242614 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.193584 31195 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.288640 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.250416 31576 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.288923 31195 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.289522 31576 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.249814 25026 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:45.186559 31476 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.291525 31576 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.291446 31476 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.259735 31469 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.289709 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.293401 31476 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.293535 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:45.249610 31325 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.294268 31325 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.259886 31299 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.222589 31488 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.257206 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.261010 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.245870 31303 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.233758 31435 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.270153 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.231218 30665 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:45.274047 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.231606 24950 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:45.233666 24947 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20240426 13:21:45.279323 25104 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:45.308992 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.222177 31540 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20240426 13:21:45.273669 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.283416 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.303673 31488 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.327597 31488 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.285012 31557 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.285650 25097 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:45.286350 31465 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.247252 31575 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.288290 31290 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.290860 25022 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:45.296277 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 73291.8us
I20240426 13:21:45.294863 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:45.296605 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 67803.8us
I20240426 13:21:45.250185 25027 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:21:45.275002 31300 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.031s
I20240426 13:21:45.271886 31513 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.303165 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:45.301913 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:45.305500 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48360.9us
I20240426 13:21:45.231920 31135 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.271057 31374 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.304018 31209 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:45.308992 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.233683 31297 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20240426 13:21:45.307179 25099 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:45.314347 25028 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:45.314560 30665 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.311837 25025 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:45.281419 31555 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20240426 13:21:45.274646 31471 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.227605 31577 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.296260 31394 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.272784 31361 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.309988 31572 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.305230 31227 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.235131 31345 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.276845 31542 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.342694 31299 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:45.334427 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70640.7us
I20240426 13:21:45.328198 25030 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:45.342940 24951 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:45.341681 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.381912 25029 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:45.382503 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.308001 31292 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.324576 31540 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:45.407130 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.384159 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.384413 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:45.385311 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.386862 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:45.322125 31303 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.290860 31195 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.310947 31587 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.383073 25103 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
I20240426 13:21:45.408504 25104 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:45.410393 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.410563 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:45.380311 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78506.6us
W20240426 13:21:45.368229 31560 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20240426 13:21:45.397001 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.422464 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:45.331692 31350 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.397414 24947 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:45.404772 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.449935 25024 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
W20240426 13:21:45.450492 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45802.4us
I20240426 13:21:45.393750 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:45.451339 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57724.5us
I20240426 13:21:45.405961 24948 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:45.383953 25029 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:21:45.360505 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.379120 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.416239 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.417510 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.411453 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:45.362167 31469 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.377647 31209 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.397055 25031 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:45.422097 25096 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [19.10-20.11]
I20240426 13:21:45.422719 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.422487 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.422575 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.423381 24954 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:45.416216 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.355058 31148 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:45.397306 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.365355 31325 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.344910 31580 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:45.295481 31432 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.352106 31541 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:45.349315 31566 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.553272 31469 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:45.553768 31469 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.554118 31469 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:45.280798 31460 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.456804 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.495186 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:45.416437 30665 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:45.414722 31572 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.492570 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.451967 31568 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:45.561710 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69218.7us
I20240426 13:21:45.497380 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.562433 25025 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:45.562716 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.497279 31558 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.526571 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:45.577390 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50899.3us
I20240426 13:21:45.415055 31465 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.410895 31497 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.407948 31308 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:21:45.455842 31570 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.532419 31361 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.522732 31446 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20240426 13:21:45.415377 31513 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.410550 31567 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.610613 31567 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.611938 31513 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.613287 31567 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:45.551477 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 154235us
I20240426 13:21:45.549243 31571 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20240426 13:21:45.407539 31540 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:45.419113 31587 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.620898 31540 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.621547 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.622078 31540 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.623026 24947 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:45.624708 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:45.627753 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:45.407202 31488 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.628582 25100 raft_consensus.cc:595] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.629094 25100 raft_consensus.cc:603] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:45.628944 31488 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.498046 31495 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20240426 13:21:45.631706 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.632460 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:45.564548 31312 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:21:45.336582 31435 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.577004 31558 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:45.455507 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 76513.4us
I20240426 13:21:45.310758 31583 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.215s
I20240426 13:21:45.524919 25103 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
W20240426 13:21:45.561362 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 160966us
I20240426 13:21:45.498229 31303 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:45.491830 31227 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:21:45.563544 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.522393 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:45.526840 31209 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.498325 31543 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:45.568478 24954 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:45.408236 31374 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.645520 31312 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.587065 31572 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.660408 31312 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:21:45.660809 31374 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.585820 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.411746 31292 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.661803 31572 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:45.662855 31292 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.408596 31557 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.663254 31135 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.648612 31308 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:45.455734 31474 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
I20240426 13:21:45.664500 31308 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.665524 25023 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:45.666309 31227 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:45.594095 31574 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20240426 13:21:45.667228 31227 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.555501 25102 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:45.669144 25618 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:45.659175 31543 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.656726 31148 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20240426 13:21:45.672006 31543 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:45.672154 31148 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:45.635489 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.636080 31369 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:45.636813 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.637575 31432 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.637578 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.639953 31460 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.641798 31577 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:45.678306 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92596.5us
I20240426 13:21:45.652922 31558 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.642959 31471 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.575372 31300 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:45.644511 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.562067 31568 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.574224 31299 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:45.696197 31299 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.696597 31299 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.646277 25025 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20240426 13:21:45.558005 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62911.2us
I20240426 13:21:45.702472 31469 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.704221 31435 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.704759 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:45.705086 25027 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:45.705670 31542 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:45.651126 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.586763 31580 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:45.706450 31542 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20240426 13:21:45.706754 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56799us
I20240426 13:21:45.707119 31568 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:45.651782 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 153012us
I20240426 13:21:45.653932 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:45.654029 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.590188 31570 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.753150 31570 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.658417 31345 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.655369 31303 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.408106 31575 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.756289 31303 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:45.757668 31575 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.672526 31148 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:45.677412 31557 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.642871 31488 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:45.678064 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:45.684062 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.659132 31513 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:45.688643 24950 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:45.656885 31571 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:45.591918 31497 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.690306 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.691102 31350 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.709872 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.710310 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:45.710661 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56790.9us
I20240426 13:21:45.711421 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:21:45.716220 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62261.1us
I20240426 13:21:45.688433 31300 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.689406 31572 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:45.753154 31325 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20240426 13:21:45.769580 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59773.6us
W20240426 13:21:45.656513 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93038.1us
I20240426 13:21:45.757469 31209 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.717617 31471 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:21:45.771747 31597 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240426 13:21:45.772287 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.772818 31587 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.765974 31350 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.773514 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.774458 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.761718 31571 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:45.776221 31135 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:45.761514 31513 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.765043 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.765254 25102 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.776860 31513 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:45.779009 25102 raft_consensus.cc:603] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:45.768817 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20240426 13:21:45.779911 31545 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:21:45.780503 31545 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:45.779861 31148 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.781479 31148 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.776688 31135 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:45.782509 31135 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.778797 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.784824 25024 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:45.778435 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.765790 31543 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240426 13:21:45.772464 31466 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:45.778000 31579 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:45.786715 31579 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.787072 31579 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:45.770468 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.789578 31585 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:45.789633 31497 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:45.790059 31585 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.777658 31514 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:45.790633 31585 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:45.790292 25622 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:45.790087 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.758569 31435 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.792728 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.793341 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:45.794446 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.799227 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.800104 25022 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:45.764847 31432 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:45.758538 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.810366 25103 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 12. Current term is 13. Ops: []
I20240426 13:21:45.810942 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:45.805150 31435 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.814348 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.814956 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.815399 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:45.768462 31300 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:45.819013 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 179) took 13092 ms (client timeout 15000 ms). Trace:
I20240426 13:21:45.819535 25023 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:45.815683 25103 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:45.816632 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.816056 25624 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:45.817993 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.820638 25024 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:45.821179 25097 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:45.818238 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:45.809958 31227 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:45.763056 31308 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20240426 13:21:45.821877 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:45.819070 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.822994 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.771095 31572 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.825282 25022 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:45.822382 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:45.837221 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:45.790223 31466 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:45.790899 31514 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:45.837277 25097 raft_consensus.cc:595] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.826905 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:45.838701 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 80231.7us
I20240426 13:21:45.828097 31578 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.810045 31469 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.813347 31557 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.840994 25097 raft_consensus.cc:603] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:45.842089 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.843595 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.844688 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.815615 31435 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.846995 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:45.817234 31603 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20240426 13:21:45.821496 31460 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:45.848875 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.850291 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.851845 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.824854 31227 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:45.819553 24852 rpcz_store.cc:269] 0426 13:21:32.720554 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:32.720750 (+   196us) service_pool.cc:225] Handling call
0426 13:21:45.812555 (+13091805us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:42.468010 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:42.468742 (+   732us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:42.468804 (+    62us) write_op.cc:427] Acquired schema lock
0426 13:21:42.468827 (+    23us) tablet.cc:660] Decoding operations
0426 13:21:42.671896 (+203069us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:42.721805 (+ 49909us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:42.726302 (+  4497us) tablet.cc:683] Acquiring locks for 14 operations
0426 13:21:42.736660 (+ 10358us) tablet.cc:699] Row locks acquired
0426 13:21:42.736697 (+    37us) write_op.cc:260] PREPARE: finished
0426 13:21:42.736882 (+   185us) write_op.cc:270] Start()
0426 13:21:42.737027 (+   145us) write_op.cc:275] Timestamp: P: 1714137702736810 usec, L: 0
0426 13:21:42.737061 (+    34us) op_driver.cc:331] REPLICATION: starting
0426 13:21:42.785574 (+ 48513us) log.cc:842] Serialized 5428 byte log entry
0426 13:21:43.069690 (+284116us) op_driver.cc:444] REPLICATION: finished
0426 13:21:43.237391 (+167701us) write_op.cc:300] APPLY: starting
0426 13:21:43.237521 (+   130us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:43.849722 (+612201us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:43.849851 (+   129us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:45.284496 (+1434645us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:45.284982 (+   486us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0426 13:21:45.285079 (+    97us) write_op.cc:311] APPLY: finished
0426 13:21:45.285615 (+   536us) log.cc:842] Serialized 129 byte log entry
0426 13:21:45.588069 (+302454us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:45.588194 (+   125us) write_op.cc:446] Released schema lock
0426 13:21:45.654528 (+ 66334us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:45.752283 (+ 97755us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:45.758390 (+  6107us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1632602,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":156135,"apply.run_cpu_time_us":82032,"apply.run_wall_time_us":2529740,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62675,"num_ops":14,"prepare.queue_time_us":80001,"prepare.run_cpu_time_us":17042,"prepare.run_wall_time_us":444454,"raft.queue_time_us":157717,"raft.run_cpu_time_us":73968,"raft.run_wall_time_us":111116,"replication_time_us":332536,"thread_start_us":674527,"threads_started":5,"wal-append.queue_time_us":462352,"wal-append.run_cpu_time_us":27874,"wal-append.run_wall_time_us":1113935}]]}
I20240426 13:21:45.842626 31572 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:45.761190 31596 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20240426 13:21:45.839998 31290 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:45.861261 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.840473 31580 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:45.869132 31312 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.864557 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.865689 30665 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:45.870312 31209 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.871059 31570 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.855034 31488 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:45.845813 31587 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:45.859647 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:45.872932 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.873589 25617 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:21:45.845422 31597 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:21:45.853075 31460 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.824487 31567 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20240426 13:21:45.845734 31557 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.858891 31227 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:45.845518 31195 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20240426 13:21:45.873011 31290 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.843830 31466 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.847998 31303 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.878600 31446 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:45.902968 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:45.852254 31578 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.904279 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:45.906011 31570 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:45.908129 31557 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:45.908478 31299 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.909793 31135 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:45.913991 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:45.898656 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:45.947702 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49119us
I20240426 13:21:45.948945 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:45.937438 25624 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:45.949493 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:45.901049 31460 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:45.950459 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:45.941778 31601 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20240426 13:21:45.941365 31227 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:45.897549 31495 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:45.897761 31490 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:21:45.930158 31600 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20240426 13:21:45.928097 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:45.954121 25031 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22.
I20240426 13:21:45.914316 31290 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:45.955406 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:45.924634 31574 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:45.915510 31101 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.959780 25617 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 21: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:45.961637 24951 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:45.972326 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.975857 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.976635 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:45.977077 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.979693 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.979962 24950 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:45.983021 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.983781 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.985982 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.947747 31465 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:45.957022 31471 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:45.957410 31543 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:45.990653 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.994300 25624 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 13.
I20240426 13:21:45.994498 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.995338 24956 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:45.991910 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:45.992575 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:45.951100 31597 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20240426 13:21:45.982964 31559 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:45.980561 31374 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:46.002837 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.009496 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.003391 24956 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240426 13:21:46.004309 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:45.914611 31514 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.013187 25023 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:46.013794 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:46.016264 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.018452 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:45.997207 30665 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:46.021994 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.028873 25615 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:46.022272 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.023151 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.024144 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.024463 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:46.035339 25618 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:46.042240 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:46.043720 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.048502 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.006212 31559 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:45.982782 31101 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.057902 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.061429 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:46.062342 24955 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:46.012888 31374 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:46.015643 31463 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:46.092415 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50269.8us
I20240426 13:21:46.093407 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.039130 31577 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:46.102670 24955 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:46.094719 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.058703 24949 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:45.997639 31465 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.112000 31465 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.006634 31513 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.116439 31513 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.117333 24950 raft_consensus.cc:2354] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:21:46.051505 31488 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:46.063624 31559 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:46.043855 31575 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:46.057729 30665 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:46.090615 31374 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:46.138149 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:46.040190 31209 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20240426 13:21:46.100926 31312 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.089958 31497 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:46.098218 31195 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:46.098459 31303 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.098096 31577 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.043524 31474 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20240426 13:21:46.146559 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:46.197912 31577 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:46.100674 31292 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:46.198657 31292 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.198994 31292 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:46.153301 25619 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20240426 13:21:46.143960 31604 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:21:46.189775 31209 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:46.206235 31209 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:46.206707 31209 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:46.207301 31292 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.209239 31292 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.205495 31497 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.213510 31497 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.133965 31432 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.146718 31575 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.232594 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:46.147859 31597 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:46.148111 31135 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.232726 31465 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:46.233289 25619 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 14.
W20240426 13:21:46.233917 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 139225us
I20240426 13:21:46.103546 31463 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.220584 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:46.256551 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.260251 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:46.222163 31605 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.244s	user 0.000s	sys 0.000s
I20240426 13:21:46.266336 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.276404 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.276641 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.222321 31466 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.222667 31299 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.278051 25619 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:46.240995 31575 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:46.228816 31432 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:46.279183 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.278203 24951 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:21:46.284622 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.313249 25621 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:46.270574 31303 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:46.235229 31578 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.285928 31622 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:21:46.292824 31101 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.293285 31292 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:46.302237 25619 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:46.305526 31463 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.267020 31465 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:46.307188 25026 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:46.235051 31227 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:46.319936 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58396.1us
I20240426 13:21:46.331213 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:46.356640 25615 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
W20240426 13:21:46.368925 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.003s	sys 0.002s
W20240426 13:21:46.369428 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.003s	sys 0.002s
I20240426 13:21:46.259299 31597 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.315378 31574 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:46.393216 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.003s	sys 0.002s
I20240426 13:21:46.394140 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:46.396842 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.003s	sys 0.002s
W20240426 13:21:46.397194 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 536782us
I20240426 13:21:46.330355 31292 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:46.370188 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 543328us
I20240426 13:21:46.404248 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:46.410831 24949 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:21:46.331907 31577 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.390463 31345 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
W20240426 13:21:46.408748 31568 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.000s
W20240426 13:21:46.414088 31350 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.641s	user 0.009s	sys 0.001s
I20240426 13:21:46.422933 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:46.423622 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:46.310073 30665 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.420464 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:46.444118 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:46.403026 31312 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:46.410418 31292 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:46.330049 31495 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
W20240426 13:21:46.417660 31583 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.000s	sys 0.010s
I20240426 13:21:46.452387 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:46.412688 31626 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
W20240426 13:21:46.453104 31579 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.000s
I20240426 13:21:46.405767 31574 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:46.482461 31148 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.700s	user 0.007s	sys 0.003s
W20240426 13:21:46.458292 31583 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.000s	sys 0.010s
W20240426 13:21:46.477454 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57208.3us
W20240426 13:21:46.459975 31558 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.756s	user 0.009s	sys 0.000s
I20240426 13:21:46.453946 31299 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:46.487170 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:46.452582 31312 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:46.506465 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 187466us
I20240426 13:21:46.434881 31568 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.453328 31350 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.434063 31545 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:46.541389 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 89449.7us
W20240426 13:21:46.550354 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 125685us
I20240426 13:21:46.443363 31571 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.546000 25026 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
W20240426 13:21:46.486490 31608 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
W20240426 13:21:46.540249 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 87945.4us
I20240426 13:21:46.487728 31287 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20240426 13:21:46.533592 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 46512.4us
I20240426 13:21:46.487524 31514 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:46.579514 24949 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20240426 13:21:46.650720 24949 pending_rounds.cc:76] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:21:46.650991 24949 pending_rounds.cc:97] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20240426 13:21:46.651214 24949 raft_consensus.cc:2866] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:46.544884 31614 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.319s	user 0.000s	sys 0.001s
I20240426 13:21:46.531426 25621 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:46.508634 31312 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:46.653390 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67597.6us
I20240426 13:21:46.659708 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:46.654814 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:46.561446 31616 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
I20240426 13:21:46.663203 25098 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:46.541736 31638 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:46.483704 31290 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:46.538106 31610 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.312s	user 0.001s	sys 0.000s
W20240426 13:21:46.548853 31617 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
W20240426 13:21:46.587471 31300 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.743s	user 0.005s	sys 0.005s
W20240426 13:21:46.653455 31611 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
I20240426 13:21:46.713428 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.559429 31636 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:46.543552 31299 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:46.728552 31374 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.000s
I20240426 13:21:46.728453 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:46.584205 31209 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:46.730422 31557 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.776s	user 0.008s	sys 0.000s
I20240426 13:21:46.730774 31209 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:46.731329 31620 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.361s	user 0.001s	sys 0.001s
I20240426 13:21:46.715965 31300 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.733287 31300 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:46.733896 31460 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.761s	user 0.010s	sys 0.000s
I20240426 13:21:46.734282 31460 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.736151 31460 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.737658 24953 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:46.730824 31557 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.740016 25622 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21.
I20240426 13:21:46.740816 31299 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.742806 31557 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.729233 31374 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.744406 25030 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:46.675957 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.745445 25025 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:46.676499 31577 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.677851 31462 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:46.679005 31446 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.525876 31631 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:46.596129 31465 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.685510 31290 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:46.687564 31559 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.001s
I20240426 13:21:46.503384 31558 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.691913 31552 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:46.759737 31634 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.248s	user 0.002s	sys 0.000s
I20240426 13:21:46.760159 31552 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.759732 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:46.761380 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.762115 31552 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.762135 25623 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:46.692308 31514 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:46.671653 31638 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:46.511652 31583 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.160s
W20240426 13:21:46.693742 31623 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20240426 13:21:46.694165 31543 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:46.751987 31631 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.699105 31312 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.625768 31287 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:46.702798 31572 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.824s	user 0.003s	sys 0.007s
I20240426 13:21:46.708051 31350 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.708148 25028 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
W20240426 13:21:46.709944 31619 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.287s	user 0.001s	sys 0.001s
I20240426 13:21:46.710386 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.757422 31465 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.715597 25615 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:21:46.744158 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:46.664021 31640 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:46.681476 31635 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:46.748399 25622 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:21:46.711088 31597 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.673228 31630 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.754297 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.759022 31559 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.758404 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:46.488639 31579 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:46.774703 25022 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.009s	sys 0.000s
I20240426 13:21:46.695078 31568 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.795080 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:46.810026 31287 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:46.768296 31514 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:46.844199 31514 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:46.763885 31300 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.848609 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:46.848367 31300 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.849704 31557 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:46.850782 31300 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.812904 25620 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:46.851264 31312 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.752589 31446 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.852360 31643 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:46.852896 31643 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.853303 31579 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.853564 31446 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.853948 24947 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20240426 13:21:46.854285 31446 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.816104 31635 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.854771 31643 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.855263 31643 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:46.854915 31446 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.776561 31543 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:46.856262 31635 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.856676 24956 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21.
I20240426 13:21:46.856773 31635 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.857290 31635 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.858971 31635 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.860433 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:46.843801 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:46.855691 31643 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.863171 25622 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:46.864347 31643 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:46.866420 31575 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.539s	user 0.005s	sys 0.003s
I20240426 13:21:46.866917 31575 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.868163 25023 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:46.811365 31574 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:46.834185 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75890.8us
I20240426 13:21:46.873955 24921 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:46.831014 25026 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
W20240426 13:21:46.834185 25022 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.010s	sys 0.000s
I20240426 13:21:46.558360 31466 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:46.764182 31432 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.777617 31631 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:46.838537 31350 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:46.842350 31559 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:46.842190 31639 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20240426 13:21:46.825012 31542 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:21:46.908512 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2456) took 1090 ms. Trace:
I20240426 13:21:46.908994 25098 rpcz_store.cc:276] 0426 13:21:45.817909 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.818025 (+   116us) service_pool.cc:225] Handling call
0426 13:21:46.908355 (+1090330us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:46.908459 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1911313664}
I20240426 13:21:46.911948 24955 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20240426 13:21:46.921438 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3268) took 1007 ms. Trace:
I20240426 13:21:46.921816 25031 rpcz_store.cc:276] 0426 13:21:45.913537 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.913755 (+   218us) service_pool.cc:225] Handling call
0426 13:21:46.921250 (+1007495us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:46.921387 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15223040}
I20240426 13:21:46.926229 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.927278 24950 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 22 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20240426 13:21:46.813764 31572 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.816457 31465 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.879431 31466 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.937258 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:46.944139 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.952714 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.953562 25099 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:46.958266 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.930883 31227 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:46.964200 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.960875 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.880407 31292 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:46.967694 31292 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.963132 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:46.968647 24951 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 21: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:46.938254 31466 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:46.838042 31630 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:46.956264 31435 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.097s	user 0.008s	sys 0.003s
I20240426 13:21:46.986704 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.987349 24950 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:46.951943 31577 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:46.962782 31618 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.457s	user 0.002s	sys 0.000s
I20240426 13:21:46.973987 31465 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:46.975451 31558 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.976069 31569 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.977597 31575 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:46.978447 31628 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.454s	user 0.002s	sys 0.005s
I20240426 13:21:46.964637 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:46.973697 31446 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:46.952666 31374 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:46.971586 31585 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.180s	user 0.013s	sys 0.000s
I20240426 13:21:47.081425 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.081996 25031 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:21:47.080940 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:46.993533 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:46.993891 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:46.994354 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.000s	sys 0.033s
W20240426 13:21:47.083400 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.000s	sys 0.033s
I20240426 13:21:46.928083 31345 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:46.992161 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:46.999176 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.040032 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.041281 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.060845 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.072190 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:47.073118 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.075260 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.076437 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.086546 25097 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:47.087342 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.087848 25097 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:47.087759 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.077430 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.077659 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.089269 25104 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:47.090082 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.071385 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:47.091334 25101 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20240426 13:21:47.095679 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.095983 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.096951 25617 raft_consensus.cc:2354] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:47.097151 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:46.996078 31557 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:47.099639 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:47.103596 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 439201us
I20240426 13:21:47.100889 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:47.102123 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 104707us
I20240426 13:21:46.982666 31227 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.117372 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.117815 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.123618 24995 leader_election.cc:398] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:21:47.124974 24995 leader_election.cc:401] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:46.990299 31435 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:46.981621 31638 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.157887 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.158435 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.158674 24951 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:47.159051 25101 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 15
I20240426 13:21:47.156174 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.162228 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:47.105676 31469 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:47.154772 25031 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:21:47.168596 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.169332 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.116676 31374 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20240426 13:21:47.172003 31292 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:21:47.172435 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:47.173045 24951 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:47.165881 31570 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.174944 31570 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.176765 31570 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.176930 31630 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.178579 31361 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:47.179059 31361 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:47.086037 31569 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.180003 31361 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:47.165699 31605 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:21:47.182826 31569 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.180142 31432 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.185482 31227 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.187626 31514 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.180285 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:47.180451 31638 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:47.111079 31557 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.199934 31557 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.180832 31572 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.201407 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3122) took 1351 ms. Trace:
I20240426 13:21:47.201802 24947 rpcz_store.cc:276] 0426 13:21:45.849685 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.849830 (+   145us) service_pool.cc:225] Handling call
0426 13:21:45.850125 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:46.854389 (+1004264us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.201169 (+346780us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540004b354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.201348 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2308135296}
I20240426 13:21:47.109310 31639 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:46.996414 31577 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.153437 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.106081 31655 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.209748 31655 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.211724 31655 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.165511 24995 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21.
I20240426 13:21:47.212203 24995 leader_election.cc:401] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:21:47.165947 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56806.7us
I20240426 13:21:47.212803 31361 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 21
I20240426 13:21:47.166687 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.141402 31495 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:47.212708 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 47345.8us
I20240426 13:21:47.201792 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.212695 25101 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:47.153906 31299 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:47.226787 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.227348 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:47.227246 31299 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.228664 24949 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:47.221045 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:47.181339 31652 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20240426 13:21:47.229841 24952 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:47.206846 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:47.224170 25026 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.224735 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.224874 31639 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.227423 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.233773 25100 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:47.194787 31514 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.239249 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.234282 24951 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:47.150074 25097 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:47.237648 24949 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:47.220348 25623 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:47.162036 31568 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.089972 31585 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.251752 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.249161 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:47.249603 25025 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:21:47.253307 25101 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20240426 13:21:47.253916 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.250994 25097 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20240426 13:21:47.267797 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:47.252636 31625 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.666s	user 0.000s	sys 0.007s
I20240426 13:21:47.288151 25031 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:47.295600 31435 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.208612 31469 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:47.283177 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69614.4us
I20240426 13:21:47.302377 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.289753 25624 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20240426 13:21:47.295231 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.300685 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.304013 25101 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:47.304664 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.266991 31497 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:47.312064 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:47.313182 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:47.314117 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.281107 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:47.314244 24947 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:47.301298 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 179691us
I20240426 13:21:47.257382 31662 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.319133 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:47.305750 31580 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.019s	sys 0.037s
I20240426 13:21:47.316118 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:47.291527 31575 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:47.330651 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.349424 25031 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:21:47.234745 31495 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.326733 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.373049 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.311152 31497 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:47.388396 25619 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:47.330775 31580 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.019s	sys 0.038s
I20240426 13:21:47.311901 31435 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.388921 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.262449 31644 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 13:21:47.268864 31568 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.399225 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.400504 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:47.403353 25101 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:47.406941 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.407909 25029 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:47.409286 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90246.5us
I20240426 13:21:47.410533 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:47.409662 31435 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.414435 31435 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.415138 31435 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:47.416800 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.417098 31568 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.417737 31568 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20240426 13:21:47.418774 31287 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.533s	user 0.009s	sys 0.000s
I20240426 13:21:47.421444 31287 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.326543 31662 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.424680 31662 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.420300 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:47.425734 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100778us
I20240426 13:21:47.395833 31497 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:47.426627 25070 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22.
I20240426 13:21:47.426983 25070 leader_election.cc:401] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20240426 13:21:47.419639 31585 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.428496 31587 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22.
I20240426 13:21:47.429630 24948 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 15: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:47.434646 31299 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.435633 31577 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.437007 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.392745 31575 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:47.452265 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:47.395159 31650 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:47.283812 31345 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.453248 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:47.462394 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:47.459798 31552 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:21:47.468221 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 304707us
I20240426 13:21:47.477623 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.478519 25031 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.469554 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.483258 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.486532 31650 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:47.487425 31650 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.488381 31663 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:47.463439 31577 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.489430 31577 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.490213 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:47.491326 31577 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.475232 31552 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.491070 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:47.492090 31552 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:47.492780 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.493355 31585 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:21:47.492807 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.493383 31497 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.499843 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.508627 25102 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:21:47.509230 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56046.1us
I20240426 13:21:47.504729 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.512230 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.514783 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.514786 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:47.519695 31662 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:47.520128 31662 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.522706 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:47.525123 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.526535 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:47.510491 31626 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.993s	user 0.002s	sys 0.026s
I20240426 13:21:47.531631 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.533311 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:47.520507 31466 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.000s
W20240426 13:21:47.532575 31633 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.011s	sys 0.018s
I20240426 13:21:47.520524 31662 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:47.514050 31585 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:47.549890 25069 leader_election.cc:398] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:47.550959 25069 leader_election.cc:401] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:47.552839 31622 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:47.555428 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.556092 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.558228 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:47.558984 31227 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:47.560642 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3129) took 1608 ms. Trace:
I20240426 13:21:47.561048 24953 rpcz_store.cc:276] 0426 13:21:45.952119 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.952296 (+   177us) service_pool.cc:225] Handling call
0426 13:21:45.974031 (+ 21735us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:46.738075 (+764044us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.560431 (+822356us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54000a7dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.560587 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1755775232}
I20240426 13:21:47.562206 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.533394 31299 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:47.536409 31626 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.002s	sys 0.027s
I20240426 13:21:47.567112 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.568112 25620 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20240426 13:21:47.524181 31639 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:47.541270 31543 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.685s	user 0.010s	sys 0.000s
I20240426 13:21:47.568634 25620 pending_rounds.cc:76] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 14
I20240426 13:21:47.568918 25102 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:21:47.537250 31630 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:47.544407 31542 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20240426 13:21:47.550464 31633 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.011s	sys 0.019s
I20240426 13:21:47.544530 31587 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.055s
I20240426 13:21:47.551144 31580 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:47.589953 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.540019 31463 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:21:47.574699 25620 pending_rounds.cc:97] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20240426 13:21:47.540241 31673 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.576535 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.567588 31567 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:47.544293 31432 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:47.547014 31572 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:47.594985 31631 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.687s	user 0.004s	sys 0.004s
I20240426 13:21:47.585563 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.589622 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:47.592062 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.582123 31676 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15.
I20240426 13:21:47.592438 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.626299 24950 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:47.578716 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.567497 31227 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:47.607178 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.607573 25620 raft_consensus.cc:2866] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:47.588336 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.537938 31568 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20240426 13:21:47.539819 31446 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20240426 13:21:47.611357 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.639041 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.575731 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:47.584777 31671 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.322s	user 0.009s	sys 0.004s
I20240426 13:21:47.567528 31577 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.541126 31466 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.584497 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.650405 25615 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.650558 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:47.651124 24947 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:47.604456 25100 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:47.652770 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.603581 31394 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:21:47.575796 31626 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:47.587474 31497 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.583246 31540 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.711354 31540 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.712188 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2640) took 1708 ms. Trace:
I20240426 13:21:47.712464 25621 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:47.712533 24956 rpcz_store.cc:276] 0426 13:21:46.004083 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:46.022873 (+ 18790us) service_pool.cc:225] Handling call
0426 13:21:47.711992 (+1689119us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:47.712138 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1901163776}
I20240426 13:21:47.713816 31540 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.602937 31639 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:47.633320 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.714731 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:47.715533 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.716252 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.716578 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.716852 24956 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:47.611685 31644 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:47.718091 31644 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.720398 31644 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.721017 31644 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:47.636394 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.634685 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.723474 25028 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:47.644549 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.646410 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:47.616576 31463 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:47.725780 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:47.725517 31463 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:47.726727 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71942.2us
I20240426 13:21:47.573791 31575 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.577318 31543 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.578276 31469 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.653084 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.656621 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.654970 31394 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:47.657923 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.634608 31227 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:47.661579 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:47.744280 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82772.7us
I20240426 13:21:47.745283 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.746341 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.746730 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:47.747516 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.662214 31466 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:47.663496 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.664676 25616 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20240426 13:21:47.748879 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:47.654620 31577 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.655303 31568 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
W20240426 13:21:47.682765 31667 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20240426 13:21:47.698158 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.706395 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.727710 24995 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21.
I20240426 13:21:47.631040 31495 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.734756 24952 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:47.621469 31574 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20240426 13:21:47.654738 31287 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.616098 31673 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.808782 25101 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 19. Current term is 20. Ops: []
W20240426 13:21:47.805913 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 1133) took 2001 ms (client timeout 2056 ms). Trace:
W20240426 13:21:47.816929 25099 rpcz_store.cc:269] 0426 13:21:45.809091 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.814416 (+  5325us) service_pool.cc:225] Handling call
0426 13:21:47.805701 (+1991285us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:47.805848 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2565283968}
I20240426 13:21:47.818284 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.805843 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3272) took 1721 ms. Trace:
I20240426 13:21:47.819934 25023 rpcz_store.cc:276] 0426 13:21:46.084705 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:46.084899 (+   194us) service_pool.cc:225] Handling call
0426 13:21:46.118422 (+ 33523us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:46.878667 (+760245us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.805648 (+926981us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.805796 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1723804544}
I20240426 13:21:47.807899 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 906) took 1807 ms. Trace:
I20240426 13:21:47.740932 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:47.823755 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82913.4us
I20240426 13:21:47.669508 24947 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:47.751036 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.631940 31660 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20240426 13:21:47.757699 24953 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:47.681488 31552 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.625254 31631 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:47.764994 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2649) took 1725 ms. Trace:
I20240426 13:21:47.849045 24955 rpcz_store.cc:276] 0426 13:21:46.039377 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:46.148811 (+109434us) service_pool.cc:225] Handling call
0426 13:21:46.149253 (+   442us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:46.912476 (+763223us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.764695 (+852219us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.764921 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1753398656}
I20240426 13:21:47.766685 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.767536 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:47.853163 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.856644 25622 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
W20240426 13:21:47.889699 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 132997us
I20240426 13:21:47.890986 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:47.750216 31227 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:47.832134 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.895354 25102 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:47.822772 25624 rpcz_store.cc:276] 0426 13:21:46.000915 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:46.001100 (+   185us) service_pool.cc:225] Handling call
0426 13:21:46.013957 (+ 12857us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:47.303733 (+1289776us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.807628 (+503895us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54007521d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.807836 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2933364096}
I20240426 13:21:47.897043 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:47.839171 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:47.842998 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 115400us
I20240426 13:21:47.729827 31644 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:47.829154 31626 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.890237 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:47.892252 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.757820 31645 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:47.750200 31587 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:47.654299 31465 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.159s
I20240426 13:21:47.768131 31446 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:47.765586 31461 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:47.829777 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78788.4us
I20240426 13:21:47.921046 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.921160 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.922025 25100 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:47.923336 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.901969 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:47.901723 25099 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.769039 31568 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:47.640462 31585 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:47.931887 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.932152 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:47.903568 25024 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.912179 31543 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:47.905710 31345 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:47.830619 31661 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20240426 13:21:47.900691 31638 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.704s	user 0.005s	sys 0.005s
W20240426 13:21:47.902208 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 195908us
I20240426 13:21:47.935833 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:47.936947 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.927760 25615 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:47.899871 31644 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:47.932214 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.941107 25102 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:47.942095 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.942718 25102 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:47.908126 31552 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:47.934718 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:47.932840 25099 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:47.830575 31655 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
W20240426 13:21:47.937625 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 2.056s (SENT) [suppressed 22 similar messages]
I20240426 13:21:47.915022 31432 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:47.899575 31580 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.828224 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.913517 31650 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:47.951434 25105 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:47.917045 31583 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:47.957717 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:47.958977 25099 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:47.962838 25100 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
W20240426 13:21:47.982017 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47386.2us
W20240426 13:21:47.984123 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52055.4us
I20240426 13:21:47.985450 25102 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:48.010283 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:48.012499 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:48.013547 25100 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:47.933851 31574 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:47.952199 31572 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.025593 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:47.935755 31345 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:47.829238 31394 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.026456 25099 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:48.031843 25100 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20240426 13:21:47.956041 31640 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:47.982576 31583 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:48.008384 31513 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:48.030412 31647 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20240426 13:21:48.026382 31557 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:48.021457 31684 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.022394 31661 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:48.030304 31652 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20240426 13:21:48.043087 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:48.052134 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122061us
I20240426 13:21:48.033895 31345 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:47.980120 31659 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.089s
I20240426 13:21:48.070401 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.045383 31583 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:48.064180 31655 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.082749 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.085844 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.051748 31676 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:47.984747 31580 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.104934 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:48.109673 24955 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:21:48.111644 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:48.118471 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.046070 31574 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20240426 13:21:48.077834 31557 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:48.084791 31650 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:48.074823 31621 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.538s	user 0.003s	sys 0.019s
I20240426 13:21:48.065184 31555 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.127166 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.129098 25104 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:48.095490 31633 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:48.104241 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:48.040500 31568 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.078758 31684 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.088829 31462 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20240426 13:21:48.075299 31374 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20240426 13:21:48.096375 31655 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:21:48.151623 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63104us
I20240426 13:21:48.159704 25615 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20240426 13:21:48.136799 31577 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:48.188109 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67873.8us
W20240426 13:21:48.215948 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88857.2us
I20240426 13:21:48.128648 31663 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.023s
I20240426 13:21:48.065868 31640 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20240426 13:21:48.143213 31673 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.217641 25029 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:48.153780 31662 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.579s	user 0.006s	sys 0.003s
I20240426 13:21:48.264637 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.259707 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3269) took 2270 ms. Trace:
I20240426 13:21:48.265738 25027 rpcz_store.cc:276] 0426 13:21:45.989272 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:45.989596 (+   324us) service_pool.cc:225] Handling call
0426 13:21:46.007056 (+ 17460us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:46.007370 (+   314us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:46.007399 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:46.007784 (+   385us) log.cc:842] Serialized 26 byte log entry
0426 13:21:46.358720 (+350936us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:46.358757 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:46.358819 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:47.874431 (+1515612us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:48.232633 (+358202us) raft_consensus.cc:1671] finished
0426 13:21:48.232665 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.259645 (+ 26980us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2368,"raft.queue_time_us":3641,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":17801,"spinlock_wait_cycles":259038976,"thread_start_us":298093,"threads_started":1,"wal-append.queue_time_us":433264}
I20240426 13:21:48.310103 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.232832 31555 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.355906 31701 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.226s
W20240426 13:21:48.370190 31693 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20240426 13:21:48.370818 31707 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:48.310168 31587 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20240426 13:21:48.370657 31677 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.298s	user 0.001s	sys 0.001s
W20240426 13:21:48.403836 31699 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
W20240426 13:21:48.404697 31689 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20240426 13:21:48.404834 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2661) took 1146 ms. Trace:
I20240426 13:21:48.405216 24951 rpcz_store.cc:276] 0426 13:21:47.258286 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.258740 (+   454us) service_pool.cc:225] Handling call
0426 13:21:47.259072 (+   332us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:47.331040 (+ 71968us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:47.331071 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:47.331577 (+   506us) log.cc:842] Serialized 26 byte log entry
0426 13:21:47.613806 (+282229us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:47.613863 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:47.808247 (+194384us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b54000a64d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:47.808310 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:48.404523 (+596213us) raft_consensus.cc:1671] finished
0426 13:21:48.404543 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.404766 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5507,"raft.queue_time_us":11553,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":26144,"spinlock_wait_cycles":151877376,"thread_start_us":281826,"threads_started":1,"wal-append.queue_time_us":501974}
I20240426 13:21:48.365866 31312 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.242923 31557 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.411146 31312 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.307212 31662 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:48.413226 31703 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20240426 13:21:48.366703 31669 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:48.441687 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2913) took 1210 ms. Trace:
I20240426 13:21:48.441996 25096 rpcz_store.cc:276] 0426 13:21:47.231307 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.231684 (+   377us) service_pool.cc:225] Handling call
0426 13:21:47.253566 (+ 21882us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:47.439517 (+185951us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:47.439562 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:47.439629 (+    67us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:47.439693 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.441499 (+1001806us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:48.441535 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.441641 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427130624}
W20240426 13:21:48.450338 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 512694us
W20240426 13:21:48.466008 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 209336us
W20240426 13:21:48.469221 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 532281us
W20240426 13:21:48.470547 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164060us
I20240426 13:21:48.471490 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.478204 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:48.472157 31361 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.259s	user 0.019s	sys 0.033s
I20240426 13:21:48.479468 31361 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21.
I20240426 13:21:48.482585 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.485144 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.480702 31580 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.483605 31630 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:48.487241 31148 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:21:48.488143 31148 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:48.488461 31148 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:48.489203 31148 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.490767 31697 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:48.491787 31684 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:48.492110 31312 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:48.493307 31694 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20240426 13:21:48.494079 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
W20240426 13:21:48.494637 31696 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20240426 13:21:48.495054 31655 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.495368 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2656) took 1330 ms. Trace:
I20240426 13:21:48.497665 31669 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:21:48.495736 24949 rpcz_store.cc:276] 0426 13:21:47.164771 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.164894 (+   123us) service_pool.cc:225] Handling call
0426 13:21:48.495224 (+1330330us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.495318 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73619840}
I20240426 13:21:48.501001 31463 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:48.502606 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.503798 31631 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:48.504599 31555 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.504873 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2897) took 1428 ms. Trace:
I20240426 13:21:48.507906 25105 rpcz_store.cc:276] 0426 13:21:47.076184 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.087556 (+ 11372us) service_pool.cc:225] Handling call
0426 13:21:48.503974 (+1416418us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b540039ec54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.504820 (+   846us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1984556288}
I20240426 13:21:48.508674 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2489) took 1510 ms. Trace:
I20240426 13:21:48.509160 25105 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:48.509780 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 926) took 1526 ms. Trace:
I20240426 13:21:48.510147 25026 rpcz_store.cc:276] 0426 13:21:46.982940 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:46.985611 (+  2671us) service_pool.cc:225] Handling call
0426 13:21:48.509622 (+1524011us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.509731 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":283532544}
I20240426 13:21:48.510128 31631 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:48.511600 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2457) took 1186 ms. Trace:
I20240426 13:21:48.512482 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.513111 24949 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:48.512874 31631 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.514014 25617 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:21:48.505753 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:21:48.514828 31631 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:48.516306 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45003.8us
I20240426 13:21:48.518121 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:48.518822 24950 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:48.521813 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:48.522985 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.524044 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.525503 31542 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:48.490427 31148 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:48.525897 31542 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:48.527113 25617 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20240426 13:21:48.527335 31542 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:48.528342 31657 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.654s	user 0.007s	sys 0.000s
I20240426 13:21:48.511952 25097 rpcz_store.cc:276] 0426 13:21:47.325610 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.325767 (+   157us) service_pool.cc:225] Handling call
0426 13:21:47.396626 (+ 70859us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:47.411341 (+ 14715us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:21:47.645251 (+233910us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.047717 (+402466us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:48.047762 (+    45us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:48.047800 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.510775 (+462975us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b180056b000. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:21:48.511401 (+   626us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b5400168854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:48.511446 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.511553 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32650,"prepare.queue_time_us":420870,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":10043,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":15382,"spinlock_wait_cycles":41077376,"thread_start_us":605217,"threads_started":2,"wal-append.queue_time_us":471830}
I20240426 13:21:48.509064 25024 rpcz_store.cc:276] 0426 13:21:47.032809 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.039694 (+  6885us) service_pool.cc:225] Handling call
0426 13:21:48.508502 (+1468808us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b54003ae8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.508622 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1876700544}
I20240426 13:21:48.533730 31660 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:48.534142 31660 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:48.534969 31660 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.539444 25617 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20240426 13:21:48.540594 31644 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.601s	user 0.019s	sys 0.000s
I20240426 13:21:48.541002 31644 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.535924 31660 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:48.545872 31568 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:48.548013 31578 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:48.550173 31638 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:48.550513 31638 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:48.551280 31394 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:21:48.552276 31545 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:48.567001 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:48.568917 25099 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:48.561758 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76694.3us
I20240426 13:21:48.576416 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.577261 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.591532 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:48.569990 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:48.594313 25618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.003s	sys 0.011s
W20240426 13:21:48.594785 25618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.003s	sys 0.011s
I20240426 13:21:48.595590 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 1069) took 1011 ms. Trace:
I20240426 13:21:48.596065 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.592377 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:48.599290 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.601831 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:48.601738 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:48.602514 25104 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:48.603503 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:48.601648 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.603668 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2494) took 1498 ms. Trace:
I20240426 13:21:48.604341 25025 rpcz_store.cc:276] 0426 13:21:47.104962 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.118962 (+ 14000us) service_pool.cc:225] Handling call
0426 13:21:47.119241 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:47.252865 (+133624us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.603458 (+1350593us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54003af2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:48.603621 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":299095808}
I20240426 13:21:48.604959 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.607301 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:48.607672 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:48.608584 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.609616 31653 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.610180 31630 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:48.610551 31630 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:48.610904 31630 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:48.611971 31577 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:21:48.613759 31716 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:48.569360 31542 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.616925 31710 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:21:48.618615 31542 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.619269 31658 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.621853 31662 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:48.622825 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:48.609691 31702 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:48.610929 25026 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:48.625370 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:48.625928 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:48.601900 31697 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:48.628631 24949 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:48.630648 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:48.634236 25615 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:21:48.599710 31578 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:48.639803 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:48.659250 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:48.662273 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:48.666332 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57812.7us
I20240426 13:21:48.667006 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:48.667596 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:21:48.668886 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45038.2us
I20240426 13:21:48.551321 31638 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.670492 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.678969 31658 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:48.668661 31702 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:48.679560 31635 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.680503 31635 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:48.680296 31702 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.680982 31635 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:48.681537 31635 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:48.684319 31698 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:48.684844 31698 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:48.685719 31698 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.694785 25098 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:48.596107 25618 rpcz_store.cc:276] 0426 13:21:47.584419 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.588253 (+  3834us) service_pool.cc:225] Handling call
0426 13:21:47.588618 (+   365us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:47.588825 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 15
0426 13:21:47.872934 (+284109us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.494052 (+621118us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:48.595193 (+101141us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:48.595265 (+    72us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:21:48.595335 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.595400 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.595533 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67854,"prepare.queue_time_us":100910,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":96016,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":11804,"spinlock_wait_cycles":16404352,"thread_start_us":884274,"threads_started":3,"wal-append.queue_time_us":357541}
I20240426 13:21:48.698292 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2500) took 1459 ms. Trace:
I20240426 13:21:48.703263 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:48.646677 31644 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:48.698763 25030 rpcz_store.cc:276] 0426 13:21:47.238771 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.238982 (+   211us) service_pool.cc:225] Handling call
0426 13:21:48.698131 (+1459149us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.698229 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2865979136}
I20240426 13:21:48.710641 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:48.727408 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52023.1us
W20240426 13:21:48.728291 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 113794us
I20240426 13:21:48.729804 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.695322 31643 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.734036 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.698202 31469 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:21:48.740504 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 475966us
I20240426 13:21:48.679623 31658 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:21:48.716645 31583 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.613s	user 0.009s	sys 0.008s
I20240426 13:21:48.744787 31466 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:48.744263 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.745841 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.707016 31674 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:48.727243 31463 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:48.773604 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:48.776441 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46764.8us
I20240426 13:21:48.732426 31325 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:48.777143 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.779772 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.781595 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:48.758484 25618 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20240426 13:21:48.727566 31290 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20240426 13:21:48.743242 31369 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.737697 31638 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:48.798035 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53854.4us
I20240426 13:21:48.781666 31662 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:48.779520 31668 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.743392 31643 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.785883 31435 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:48.783214 31325 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:48.775594 31545 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20240426 13:21:48.795336 31697 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:21:48.803153 31361 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:48.795037 31580 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:48.774014 31674 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.780252 31463 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.774500 31583 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.804337 31312 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:48.842983 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:48.855904 25097 raft_consensus.cc:595] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.856338 25097 raft_consensus.cc:603] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:48.856375 31674 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:48.803490 31394 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:48.857575 31446 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:48.857956 31446 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:48.858703 31446 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.809725 31711 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:48.844321 31303 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20240426 13:21:48.862442 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2468) took 1206 ms. Trace:
I20240426 13:21:48.868386 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:48.869103 25097 raft_consensus.cc:595] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.869480 25097 raft_consensus.cc:603] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:48.810230 31292 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:48.862900 25103 rpcz_store.cc:276] 0426 13:21:47.656106 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.044196 (+388090us) service_pool.cc:225] Handling call
0426 13:21:48.506086 (+461890us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:48.506258 (+   172us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:48.506521 (+   263us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.795089 (+288568us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.803118 (+  8029us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:48.803161 (+    43us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:48.803197 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.803249 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.862245 (+ 58996us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b5400169c50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.862376 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13081,"prepare.queue_time_us":1460,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":53,"raft.queue_time_us":17631,"raft.run_wall_time_us":50759,"spinlock_wait_cycles":1123016192,"thread_start_us":281481,"threads_started":1}
I20240426 13:21:48.874294 31633 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:48.874845 31633 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:48.875531 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 2892) took 1800 ms (client timeout 1551 ms). Trace:
I20240426 13:21:48.846567 25027 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
W20240426 13:21:48.875932 25098 rpcz_store.cc:269] 0426 13:21:47.074760 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.074956 (+   196us) service_pool.cc:225] Handling call
0426 13:21:48.875380 (+1800424us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:48.875472 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3698869120}
I20240426 13:21:48.873859 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:48.887538 25097 raft_consensus.cc:595] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.889251 31701 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:48.888248 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44354.7us
I20240426 13:21:48.888651 31571 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:21:48.889746 31701 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.891535 31701 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:48.892184 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44401.6us
I20240426 13:21:48.894579 31361 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:48.875871 31633 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.835822 31557 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.895495 31361 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.872494 31292 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.906939 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.897096 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:48.859925 31446 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:48.909106 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.907279 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:48.911963 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:48.912493 25101 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:48.913941 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2477) took 1255 ms. Trace:
I20240426 13:21:48.796379 31369 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:48.908017 31722 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20240426 13:21:48.877988 31462 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:48.905508 31633 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:48.882378 31227 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.967s	user 0.028s	sys 0.024s
I20240426 13:21:48.918784 31461 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:48.931684 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
W20240426 13:21:48.905447 31575 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.016s	sys 0.017s
I20240426 13:21:48.916397 31640 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:48.933753 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:48.934813 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 206351us
I20240426 13:21:48.934890 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.938266 24949 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
W20240426 13:21:48.935659 31575 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.016s	sys 0.017s
I20240426 13:21:48.850111 31658 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.945474 25098 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:21:48.943709 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:48.907953 31557 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:48.953253 25028 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 14. Current term is 16. Ops: []
I20240426 13:21:48.953269 25030 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:21:48.955191 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.955935 25103 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:21:48.927515 31462 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:48.936080 31706 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:48.963246 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:48.936489 31640 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:48.947052 31303 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:48.991216 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:48.990479 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:48.967631 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:48.945245 31713 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:48.963989 31290 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:49.007807 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.005537 30665 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240426 13:21:49.005398 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:49.036403 25103 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:48.967273 31542 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:48.964499 31463 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.008323 31303 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:49.055287 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.056109 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:49.057063 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44698.1us
I20240426 13:21:48.937047 31461 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:48.934629 31227 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.068327 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:49.058570 25103 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:49.107228 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:48.995187 31674 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:49.007752 31290 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.108170 25103 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:49.130410 24954 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20240426 13:21:49.108317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50363.6us
I20240426 13:21:49.058176 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:49.123603 31568 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.577s	user 0.008s	sys 0.000s
I20240426 13:21:49.059322 31303 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:49.008136 31706 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:49.121639 31724 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20240426 13:21:49.008075 31651 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:49.114408 25100 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20240426 13:21:49.005844 31713 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:49.116541 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3288) took 1537 ms. Trace:
I20240426 13:21:49.114305 31646 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.126353 25104 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 15. Current term is 16. Ops: []
W20240426 13:21:49.057490 31345 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.961s	user 0.016s	sys 0.008s
I20240426 13:21:49.135524 31622 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:49.136346 31714 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:49.136891 31557 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.129139 31674 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.139928 25618 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:21:49.140262 25105 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20240426 13:21:49.165513 25105 pending_rounds.cc:76] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 14
I20240426 13:21:49.165779 25105 pending_rounds.cc:97] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20240426 13:21:49.166110 25105 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:49.141544 25031 rpcz_store.cc:276] 0426 13:21:47.579293 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.717362 (+138069us) service_pool.cc:225] Handling call
0426 13:21:47.717634 (+   272us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:47.741813 (+ 24179us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:47.741854 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:47.748360 (+  6506us) log.cc:842] Serialized 26 byte log entry
0426 13:21:48.543507 (+795147us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b180041f8c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:48.804614 (+261107us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:48.804653 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.804720 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:49.116256 (+311536us) raft_consensus.cc:1671] finished
0426 13:21:49.116278 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:49.116484 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23809,"raft.queue_time_us":25452,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":54586,"spinlock_wait_cycles":7121792,"thread_start_us":260958,"threads_started":1,"wal-append.queue_time_us":1242150}
I20240426 13:21:49.143626 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.049116 31736 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:49.145555 25102 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:21:49.128996 31429 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.146730 31698 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:49.147907 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89796.1us
W20240426 13:21:49.125516 31723 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20240426 13:21:49.107391 31710 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.134228 31290 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:49.168146 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.187817 24951 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:21:49.168812 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:49.190258 25104 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:21:49.191083 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1390) took 1072 ms. Trace:
I20240426 13:21:49.191682 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:49.171887 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:49.109474 31626 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20240426 13:21:49.181937 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:49.144624 31706 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:49.144789 31713 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:49.156409 31583 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.191509 25624 rpcz_store.cc:276] 0426 13:21:48.119117 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.119287 (+   170us) service_pool.cc:225] Handling call
0426 13:21:48.127154 (+  7867us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:48.479229 (+352075us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:48.479295 (+    66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:48.479875 (+   580us) log.cc:842] Serialized 26 byte log entry
0426 13:21:48.777737 (+297862us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:48.777924 (+   187us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:49.164665 (+386741us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:49.164715 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:49.190826 (+ 26111us) raft_consensus.cc:1671] finished
0426 13:21:49.190850 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:49.191031 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":103248,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":310147456,"thread_start_us":514181,"threads_started":2,"wal-append.queue_time_us":451684}
W20240426 13:21:49.315212 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 133359us
I20240426 13:21:49.160398 31345 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.154228 31646 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.147583 31651 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.363976 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:49.379419 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59093.8us
I20240426 13:21:49.189708 31292 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.383502 25104 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:49.111673 31462 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.194128 31736 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:49.314013 31714 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20240426 13:21:49.363962 31630 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.752s	user 0.010s	sys 0.000s
I20240426 13:21:49.349529 31716 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:49.398780 31580 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:49.400291 31542 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.393738 31101 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.008s
I20240426 13:21:49.405655 31325 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.415097 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2474) took 1757 ms. Trace:
I20240426 13:21:49.493297 25100 rpcz_store.cc:276] 0426 13:21:47.657836 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.478079 (+820243us) service_pool.cc:225] Handling call
0426 13:21:48.478416 (+   337us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:49.141041 (+662625us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:49.414803 (+273762us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:49.415018 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1461882752}
W20240426 13:21:49.471777 31737 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.325s	user 0.000s	sys 0.001s
W20240426 13:21:49.420250 31730 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
W20240426 13:21:49.421993 31738 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:21:49.422760 31728 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.282s	user 0.000s	sys 0.000s
W20240426 13:21:49.423555 31733 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.364s	user 0.001s	sys 0.000s
I20240426 13:21:49.187296 31429 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.424209 31435 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20240426 13:21:49.425438 31727 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.335s	user 0.001s	sys 0.000s
I20240426 13:21:49.425902 31462 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:49.427160 31651 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.433584 31646 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:49.435156 31695 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.089s	user 0.005s	sys 0.000s
I20240426 13:21:49.436759 31736 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.362370 31707 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:49.349674 31369 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.162051 31640 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.313589 31710 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.383073 31650 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:21:49.484934 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 704460us
I20240426 13:21:49.347328 31668 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.096s
W20240426 13:21:49.419370 31735 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.258s	user 0.000s	sys 0.000s
I20240426 13:21:49.480088 31394 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.555141 31394 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.557561 31394 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.552457 31312 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:49.559265 31312 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:49.540434 31227 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.563621 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2518) took 1829 ms. Trace:
I20240426 13:21:49.566254 25022 rpcz_store.cc:276] 0426 13:21:47.734220 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.796892 (+ 62672us) service_pool.cc:225] Handling call
0426 13:21:47.797345 (+   453us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:47.901908 (+104563us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:47.902451 (+   543us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:47.905162 (+  2711us) log.cc:842] Serialized 26 byte log entry
0426 13:21:48.033029 (+127867us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:48.033068 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.033132 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:49.434547 (+1401415us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:49.563334 (+128787us) raft_consensus.cc:1671] finished
0426 13:21:49.563361 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:49.563562 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104128,"raft.queue_time_us":120152,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":9617,"spinlock_wait_cycles":115409152,"thread_start_us":127599,"threads_started":1,"wal-append.queue_time_us":402202}
I20240426 13:21:49.474336 31714 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:49.476903 31630 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:49.541404 31446 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.623s	user 0.006s	sys 0.005s
I20240426 13:21:49.572388 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3298) took 1915 ms. Trace:
I20240426 13:21:49.538511 31325 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:49.495630 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:21:49.564929 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1395) took 1014 ms. Trace:
I20240426 13:21:49.570442 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.570926 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:49.483103 31580 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.572862 25028 rpcz_store.cc:276] 0426 13:21:47.657229 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.724813 (+ 67584us) service_pool.cc:225] Handling call
0426 13:21:47.725065 (+   252us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:48.962528 (+1237463us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:49.572165 (+609637us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54002f9c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:49.572331 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2824228608}
I20240426 13:21:49.555804 31710 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.597152 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:49.597636 25104 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:21:49.576507 25617 rpcz_store.cc:276] 0426 13:21:48.550051 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.550170 (+   119us) service_pool.cc:225] Handling call
0426 13:21:48.550402 (+   232us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:48.550647 (+   245us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:48.550671 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:48.551066 (+   395us) log.cc:842] Serialized 26 byte log entry
0426 13:21:48.795650 (+244584us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:48.795687 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.795765 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:49.564679 (+768914us) raft_consensus.cc:1671] finished
0426 13:21:49.564708 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:49.564863 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":813,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":244263,"threads_started":1,"wal-append.queue_time_us":394439}
I20240426 13:21:49.495055 31542 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.610325 31557 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:21:49.611908 31345 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.568832 31674 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.617678 31706 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.618806 25025 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:49.619695 31706 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:48.915042 25096 rpcz_store.cc:276] 0426 13:21:47.658615 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.488727 (+830112us) service_pool.cc:225] Handling call
0426 13:21:48.488996 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:48.489172 (+   176us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:48.489440 (+   268us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.913412 (+423972us) log.cc:842] Serialized 14 byte log entry
0426 13:21:48.913602 (+   190us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:48.913655 (+    53us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:48.913708 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:48.913783 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:48.913891 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17770,"prepare.queue_time_us":64370,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":34,"raft.queue_time_us":34895,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":323314,"threads_started":1}
I20240426 13:21:49.622258 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:49.581508 31714 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:21:49.585556 31394 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:49.596225 31660 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:49.636483 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:49.591194 31668 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:49.639475 31542 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.640372 31570 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:49.640906 31639 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:49.637930 31635 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.956s	user 0.016s	sys 0.000s
W20240426 13:21:49.644222 31303 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.508s	user 0.004s	sys 0.007s
W20240426 13:21:49.644294 31740 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20240426 13:21:49.636025 31713 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.645825 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:49.646088 31580 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.649549 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.670605 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:21:49.591830 31585 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20240426 13:21:49.654124 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:49.643136 31633 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.709s	user 0.007s	sys 0.004s
I20240426 13:21:49.655825 31635 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.655308 31639 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.654300 31570 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:49.661763 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 473206us
I20240426 13:21:49.661319 31303 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.666344 31713 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.666785 25030 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:49.652750 31638 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:49.668476 31429 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.673197 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:49.674235 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:49.677006 25097 raft_consensus.cc:595] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:49.652591 31668 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:49.684882 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 304526us
I20240426 13:21:49.649034 31394 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.696235 25097 raft_consensus.cc:603] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:49.677870 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:49.714864 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:49.715703 25098 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:49.696929 25104 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:49.717514 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:49.727301 24955 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:49.729524 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:49.732216 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:49.725849 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:49.736068 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.740643 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:49.741968 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.742689 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:49.745915 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:49.746228 24956 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20240426 13:21:49.748378 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.725580 31429 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:49.754145 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:49.760537 25619 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:49.756824 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:49.725752 31638 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:49.725926 31580 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.778373 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:21:49.779481 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:49.741449 31572 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:49.783625 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:49.752823 31558 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:49.753443 31575 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.750970 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:49.790342 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:49.791544 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:49.785708 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:49.758344 31429 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:49.758437 31630 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.795899 25104 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:49.809689 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.797195 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.816112 25104 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:49.751849 31345 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20240426 13:21:49.818940 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:49.822742 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.768640 31651 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20240426 13:21:49.823458 25027 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:21:49.788760 31558 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.829382 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:49.836431 25028 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:49.837348 31635 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.837409 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:49.838515 31742 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20240426 13:21:49.838655 31345 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:49.839196 31345 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:49.841022 25103 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20240426 13:21:49.840246 31345 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:49.842185 31748 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:21:49.798281 31570 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.810679 31429 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:49.842077 31345 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:49.812654 31713 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.810441 31674 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:49.783046 31580 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.866195 31706 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:21:49.831333 31583 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:49.824406 31660 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:49.885855 31660 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.886849 31631 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:49.887384 31631 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:49.847538 31558 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:49.800453 31575 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.888370 31631 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.889451 31575 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.876217 31713 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.889874 31631 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:49.891248 31713 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.884915 31668 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:49.885299 31290 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.000s
I20240426 13:21:49.892241 31290 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.894131 31290 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.887873 31660 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.896462 31394 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.897281 31558 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:49.900738 31558 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:49.874680 31723 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240426 13:21:49.861039 31635 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:49.904745 31303 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.907541 31639 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:49.928185 31580 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.908169 31635 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:49.930349 31635 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.934428 31461 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:49.905627 31674 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.011480 31674 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:49.911199 31714 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.014357 31714 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:49.792150 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 477827us
I20240426 13:21:50.069744 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.074340 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:49.845080 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:50.082051 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 236933us
I20240426 13:21:50.083142 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.085237 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.086217 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.086620 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.087906 25022 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:50.105401 31633 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:21:50.113507 31325 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.201102 31580 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.205997 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2529) took 1001 ms. Trace:
I20240426 13:21:50.211436 31766 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.212582 31766 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.215615 31707 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:50.101068 31513 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.216598 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2686) took 1503 ms. Trace:
I20240426 13:21:50.216737 31513 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.217135 24949 rpcz_store.cc:276] 0426 13:21:48.712617 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.712786 (+   169us) service_pool.cc:225] Handling call
0426 13:21:48.736428 (+ 23642us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:48.943457 (+207029us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.216135 (+1272678us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.216401 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463739648}
I20240426 13:21:50.219327 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3158) took 2313 ms. Trace:
I20240426 13:21:50.219782 24954 rpcz_store.cc:276] 0426 13:21:47.905318 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:47.905491 (+   173us) service_pool.cc:225] Handling call
0426 13:21:47.933324 (+ 27833us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:49.130839 (+1197515us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.218800 (+1087961us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000a69d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.219129 (+   329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2752563584}
I20240426 13:21:50.220908 31462 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.221745 31462 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.202091 31580 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.233526 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2521) took 1293 ms. Trace:
I20240426 13:21:50.233952 25025 rpcz_store.cc:276] 0426 13:21:48.940423 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.940578 (+   155us) service_pool.cc:225] Handling call
0426 13:21:48.940843 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:49.619131 (+678288us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.233011 (+613880us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.233340 (+   329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1558741760}
I20240426 13:21:50.235395 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 934) took 1268 ms. Trace:
I20240426 13:21:50.235798 25030 rpcz_store.cc:276] 0426 13:21:48.967273 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.967391 (+   118us) service_pool.cc:225] Handling call
0426 13:21:48.967624 (+   233us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:49.693234 (+725610us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.234965 (+541731us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54000564d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.235235 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1607565568}
I20240426 13:21:50.234081 31580 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.243340 31462 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:50.246837 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2472) took 2589 ms (client timeout 3307 ms). Trace:
I20240426 13:21:50.249269 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2482) took 2589 ms. Trace:
W20240426 13:21:50.247435 25102 rpcz_store.cc:269] 0426 13:21:47.657266 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.010035 (+352769us) service_pool.cc:225] Handling call
0426 13:21:50.246253 (+2236218us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:50.246547 (+   294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2608490240}
I20240426 13:21:50.249786 25101 rpcz_store.cc:276] 0426 13:21:47.659891 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.497515 (+837624us) service_pool.cc:225] Handling call
0426 13:21:48.497767 (+   252us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:48.914258 (+416491us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.248653 (+1334395us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b54003ef8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.249033 (+   380us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":950545152}
I20240426 13:21:50.252451 31763 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.253556 31763 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.258766 31575 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:50.259465 31575 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:50.261014 31575 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.233393 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:50.263790 31462 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:50.264847 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.266534 31769 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:50.266942 31769 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.267324 31769 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.268162 31446 raft_consensus.cc:979] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:50.268065 31394 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.268683 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.269483 25102 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.269011 31394 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.270175 31446 raft_consensus.cc:1067] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:21:50.271821 31394 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.272264 31513 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.206621 25031 rpcz_store.cc:276] 0426 13:21:49.204074 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:49.204417 (+   343us) service_pool.cc:225] Handling call
0426 13:21:49.204799 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:49.348359 (+143560us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:49.348390 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:49.348857 (+   467us) log.cc:842] Serialized 26 byte log entry
0426 13:21:49.562313 (+213456us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:49.791976 (+229663us) log.cc:842] Serialized 14 byte log entry
0426 13:21:50.205167 (+413191us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.205382 (+   215us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:50.205454 (+    72us) raft_consensus.cc:1671] finished
0426 13:21:50.205491 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:50.205774 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":189404,"prepare.queue_time_us":314679,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":184698,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":17800,"thread_start_us":751861,"threads_started":3,"wal-append.queue_time_us":364502}
I20240426 13:21:50.276439 25028 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:50.194314 25024 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:21:50.249739 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 2925) took 1294 ms. Trace:
I20240426 13:21:50.197834 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.283309 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.284282 25096 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:50.284543 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.285050 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.285286 25101 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:50.285837 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:21:50.281098 25105 rpcz_store.cc:276] 0426 13:21:48.954796 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.956561 (+  1765us) service_pool.cc:225] Handling call
0426 13:21:48.956811 (+   250us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:49.166525 (+209714us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.249182 (+1082657us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.249547 (+   365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":421125504}
W20240426 13:21:50.287870 31312 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.728s	user 0.010s	sys 0.007s
I20240426 13:21:50.288358 25102 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:50.288545 31312 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.243480 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 75) took 1317 ms. Trace:
I20240426 13:21:50.305227 31639 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.305297 31716 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.306129 31639 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.306313 31716 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.305063 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50528 (request call id 77) took 1116 ms. Trace:
I20240426 13:21:50.307519 25623 rpcz_store.cc:276] 0426 13:21:49.147675 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:49.147933 (+   258us) service_pool.cc:225] Handling call
0426 13:21:50.261996 (+1114063us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:50.263856 (+  1860us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39189248}
I20240426 13:21:50.308317 31630 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:21:50.310829 31461 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.311589 31461 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.314321 25105 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:50.317112 25103 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:50.318193 31626 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:50.318763 31626 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:50.318887 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.319865 25096 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:50.304446 25097 rpcz_store.cc:276] 0426 13:21:48.925907 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.926046 (+   139us) service_pool.cc:225] Handling call
0426 13:21:50.242971 (+1316925us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:50.243180 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1711352192}
I20240426 13:21:50.331626 31635 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.332496 31635 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.335196 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.344368 25105 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:50.329967 31580 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.345691 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:21:50.347116 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97067.9us
I20240426 13:21:50.358805 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2492) took 1076 ms. Trace:
I20240426 13:21:50.360005 31312 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.346050 31633 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.347465 31763 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.367116 31583 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.355657 31639 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.368461 31774 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.369028 31702 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:50.323688 31626 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.370344 31746 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.370832 25617 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:50.371251 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.374624 31650 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:50.374707 31641 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:50.373483 31701 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
W20240426 13:21:50.375661 31740 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20240426 13:21:50.380470 31716 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.382755 31721 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:50.412878 31721 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:21:50.384538 31580 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.385706 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.367530 31461 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.386540 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.389158 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.395192 31626 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:50.418062 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.420162 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.413268 31721 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:50.416435 25619 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:21:50.416860 25097 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:21:50.420344 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.359249 25100 rpcz_store.cc:276] 0426 13:21:49.203687 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:49.588529 (+384842us) service_pool.cc:225] Handling call
0426 13:21:49.588833 (+   304us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:49.603118 (+ 14285us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:49.603163 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:49.610822 (+  7659us) log.cc:842] Serialized 26 byte log entry
0426 13:21:49.804453 (+193631us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:49.804509 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:49.804598 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:50.109104 (+304506us) raft_consensus.cc:1671] finished
0426 13:21:50.109164 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:50.280407 (+171243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20707,"raft.queue_time_us":40198,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10765,"thread_start_us":193253,"threads_started":1,"wal-append.queue_time_us":397946}
I20240426 13:21:50.424921 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.427930 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:50.428273 31461 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:50.428710 25100 raft_consensus.cc:595] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.429106 25100 raft_consensus.cc:603] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:21:50.429219 31429 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.012s
I20240426 13:21:50.429698 31429 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.430908 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:21:50.430852 25097 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 25.
I20240426 13:21:50.431668 31429 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.434650 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.435524 24954 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:50.436666 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.437454 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:50.438596 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.438810 31746 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.439297 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:21:50.441166 31746 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.443576 31643 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.448796 31640 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:50.449393 31583 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.451597 31583 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.452139 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.456826 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 1236) took 1289 ms. Trace:
I20240426 13:21:50.456940 31685 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.457293 24947 rpcz_store.cc:276] 0426 13:21:49.167663 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:49.167828 (+   165us) service_pool.cc:225] Handling call
0426 13:21:50.456479 (+1288651us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:50.456657 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2025187712}
I20240426 13:21:50.457535 31685 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.459688 31685 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.441358 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:50.485697 25103 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:50.465317 31721 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.487089 31766 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.549800 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.482800 31650 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.474558 25023 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:50.490504 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.341773 31325 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.487367 31580 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.407490 31641 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.484311 31633 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.474992 31774 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.488175 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.481639 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2489) took 1531 ms. Trace:
I20240426 13:21:50.432701 25102 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:50.516398 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.516739 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.522022 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.529187 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.525887 31292 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.565368 25029 raft_consensus.cc:2396] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20240426 13:21:50.535444 25097 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:50.535965 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.566351 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.566660 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.566977 25025 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:50.567523 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.563351 25098 rpcz_store.cc:276] 0426 13:21:48.950501 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:48.950646 (+   145us) service_pool.cc:225] Handling call
0426 13:21:48.950879 (+   233us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:49.716412 (+765533us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:50.481250 (+764838us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:50.481486 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758448000}
I20240426 13:21:50.567873 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.568151 31707 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.537571 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.538669 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.540263 31674 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:50.568655 31662 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.568737 31707 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.569386 31662 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.540678 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.538408 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.547719 31635 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.541285 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:50.571164 31631 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.672s	user 0.016s	sys 0.006s
I20240426 13:21:50.551004 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:21:50.551709 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.565456 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.556177 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.557569 31721 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:50.558403 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.558909 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.551949 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.565014 31292 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.528420 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:50.565493 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.567751 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:50.570437 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.532321 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.472983 31702 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.571161 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.568657 25025 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:50.573805 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58767.5us
I20240426 13:21:50.544466 31769 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.575331 25022 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:50.579517 31674 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:50.585124 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.586241 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.586256 25617 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:50.584069 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:50.588806 31778 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20240426 13:21:50.589519 24947 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:50.580539 31635 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:21:50.591598 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 245395us
I20240426 13:21:50.591714 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.592918 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.593130 31720 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20240426 13:21:50.593154 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61283.3us
I20240426 13:21:50.593745 31720 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.594210 31720 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.594983 31290 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:50.595220 31706 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:50.597782 31292 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.599085 31633 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.600869 31641 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.602391 31774 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.587494 31674 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:50.603147 31720 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.604562 31325 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.604661 31674 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.605921 31720 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.606241 31674 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.607606 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.607874 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.583653 31626 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:50.608366 24952 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:50.611317 24948 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:21:50.611743 24948 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:21:50.612136 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 937) took 1608 ms (client timeout 1856 ms). Trace:
W20240426 13:21:50.612532 25029 rpcz_store.cc:269] 0426 13:21:49.003857 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:49.007041 (+  3184us) service_pool.cc:225] Handling call
0426 13:21:50.611899 (+1604858us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:50.612070 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3355558784}
I20240426 13:21:50.612885 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.613821 24953 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:21:50.606309 31345 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.741s	user 0.009s	sys 0.010s
I20240426 13:21:50.613720 24848 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 15, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 15 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:50.608433 24950 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:50.590143 25618 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:50.617722 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.618139 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:50.624291 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43670.9us
I20240426 13:21:50.625654 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.627198 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.592279 31580 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:50.628290 24947 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:50.628427 31292 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:21:50.630734 31292 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:50.631150 31292 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:21:50.631237 31707 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.617353 31626 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.629405 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.636551 31626 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:50.591291 31635 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.641038 24948 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.642357 31633 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:21:50.642961 31633 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:50.644141 31633 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.641188 31580 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:50.646502 31580 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.640646 31641 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:50.647404 31641 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:50.647894 31641 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:50.649480 31662 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.653618 25104 raft_consensus.cc:2396] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20240426 13:21:50.647148 31580 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:50.643985 24848 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 16, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 16 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:50.657158 31702 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.657755 31635 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:50.661190 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.666921 25104 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:50.669669 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.670442 24951 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:50.671057 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.672348 31650 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.672806 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.673764 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.676633 31658 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.677181 31662 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.677151 31658 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.677950 31662 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.678879 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:50.679284 31658 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.680415 25617 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:50.678897 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:50.684171 31721 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.684299 31643 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.684691 31721 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.684829 31643 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.686802 31643 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:50.687216 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46203.1us
I20240426 13:21:50.688045 31596 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:50.688496 31596 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.688822 31596 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:50.689046 31769 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.690997 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.691900 31461 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.692236 31583 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:50.692498 31461 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.694202 31662 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.694692 31461 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.694898 31658 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.695559 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.696213 24950 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
W20240426 13:21:50.696846 31783 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:21:50.697072 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.697504 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.697733 24954 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:50.697999 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.698827 31568 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.699357 31568 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.665082 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.700969 25101 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:50.701308 31568 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.618885 31325 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.702211 31461 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:50.702060 31325 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.702556 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.703358 31461 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:21:50.703855 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.704715 31461 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:50.705653 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:50.705930 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 125714us
I20240426 13:21:50.706295 31702 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:21:50.707161 31461 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:50.695411 31658 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.713599 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.716282 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.706929 31774 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:21:50.720821 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.722028 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.722191 24948 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.723610 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.722008 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.721558 25023 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:50.736363 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:50.736554 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.728221 31461 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:50.632018 31707 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.096s
I20240426 13:21:50.732854 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:50.737676 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 75663.1us
I20240426 13:21:50.738413 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.752916 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.750062 31662 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:50.687819 31641 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.748317 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.737783 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.757295 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.757387 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.781536 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.760624 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:50.783485 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49209.5us
I20240426 13:21:50.781038 24954 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:50.784281 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.728143 31643 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:50.783177 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:50.786307 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:50.788362 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53834us
I20240426 13:21:50.752537 31753 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20240426 13:21:50.788362 30665 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:21:50.789796 31753 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:50.790238 31753 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:50.789583 31639 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:50.787907 31227 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:50.782855 25029 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:50.760174 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.687197 31721 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.801398 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.805135 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:50.756515 31394 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:50.792476 25026 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:50.737879 31769 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:50.812186 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:50.752198 31325 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.824136 31707 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:50.824594 24848 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 14, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:50.825937 25022 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:50.828428 25027 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 20 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20240426 13:21:50.670424 31720 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.132s
I20240426 13:21:50.756287 31596 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.800812 31227 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:50.834255 31658 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.790956 31639 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.837229 31583 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:50.839371 31641 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.814843 31662 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:50.806561 31643 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20240426 13:21:50.847142 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43799.1us
I20240426 13:21:50.844460 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.853610 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.855357 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.759136 31702 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.082s
I20240426 13:21:50.861474 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.813925 31429 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.864161 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.868616 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.858750 31325 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:50.860102 31568 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:50.860163 31639 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:50.859917 31583 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.881171 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.864380 31662 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.889477 25099 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20240426 13:21:50.890784 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:50.890977 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.891721 30665 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:50.892112 31290 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:50.895677 31747 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:50.896265 31747 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:50.897243 31747 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.901528 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:50.902916 30665 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.903394 30665 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:50.864784 31643 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.908282 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:50.891491 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 131098us
I20240426 13:21:50.867228 31769 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.876477 31641 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:50.914165 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:50.879086 31325 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.807052 31753 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.881656 31583 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:50.901468 31662 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:50.884604 31394 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:21:50.926251 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107275us
I20240426 13:21:50.881289 31721 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:50.911221 31643 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:50.874317 31626 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.922206 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.926023 31630 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20240426 13:21:50.929491 31714 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:50.937623 31714 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:50.938170 31707 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:50.938872 31707 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.932165 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.937942 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.941185 25026 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.900894 31227 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.943022 31227 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:50.944450 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:50.945272 31707 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.945495 31568 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.945938 31568 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:50.946171 31658 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:50.946904 31658 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:50.946142 24848 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 11, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 11 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:50.949029 31596 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:50.951962 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.955760 25026 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:21:50.959170 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:50.959903 25026 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:50.921169 31292 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.937644 31753 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:50.964665 31292 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:50.922186 31325 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:50.879406 31429 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.970237 31429 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.970402 25026 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:50.971217 25618 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:50.972254 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:50.975123 31751 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.975666 31751 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.977048 31626 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:50.978195 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44943.3us
W20240426 13:21:50.979573 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 115023us
I20240426 13:21:50.919458 31641 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:50.991109 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:50.993223 31630 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:21:50.993587 31514 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:50.994396 31514 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:50.995285 31514 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:50.996557 31514 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:50.997205 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.997905 25103 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:21:50.998665 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:50.999179 25103 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:51.000214 31736 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.000764 31736 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.002741 31736 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.003196 31736 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:50.930934 31769 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.010689 31769 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.010912 31580 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.012542 25098 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.014168 31580 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.014782 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.015431 25029 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:50.995782 31751 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:50.927251 31763 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:50.986712 31641 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:51.022346 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:51.023350 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.024197 31763 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:51.024695 31763 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.025588 31763 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:50.930691 31721 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:51.029562 31721 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:51.030584 25617 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20240426 13:21:51.033120 31711 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:51.033542 31711 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:51.034384 31711 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.038887 31640 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.039449 31640 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.041672 31640 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.043138 31635 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:51.045567 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58217.5us
I20240426 13:21:51.055720 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:50.938030 31714 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:51.060595 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:50.987766 31394 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:50.987622 31461 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:51.035904 31711 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.066274 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.003672 31736 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.070530 25617 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:51.071432 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.073575 25099 raft_consensus.cc:2354] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:51.078959 24996 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:21:51.079255 24996 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20240426 13:21:51.080721 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.072724 25623 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:51.044155 31325 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.090613 24954 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:51.091612 25618 raft_consensus.cc:2396] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20240426 13:21:51.091958 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:50.987339 31290 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.082s
W20240426 13:21:51.096879 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 114042us
I20240426 13:21:51.103272 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.104193 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:51.104928 25617 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:51.106303 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.108459 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.106734 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.109853 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.110000 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.112365 24947 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:51.080062 31720 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:21:51.111660 24956 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:51.074455 31394 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.115200 24954 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:51.117280 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.118273 24949 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:51.065364 31639 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.116415 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:51.108043 31702 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:51.129666 24996 leader_election.cc:398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:51.131225 24996 leader_election.cc:401] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20240426 13:21:51.129330 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:51.135639 31785 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.142s	user 0.003s	sys 0.000s
I20240426 13:21:51.144824 25102 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.144755 24995 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:21:51.126672 31769 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:21:51.149435 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.127290 31720 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:51.148358 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.127604 24848 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 14, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.141099 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.131158 31702 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:51.136873 31640 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:51.160401 24848 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 13, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 13 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:21:51.137398 31461 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.163931 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.167687 25029 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20240426 13:21:51.157299 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.136384 31292 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:51.171061 25104 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:51.177074 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:21:51.179505 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.179749 25026 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:51.185909 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.186231 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.155834 31631 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:51.188503 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.190022 31736 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.189555 31720 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.166517 31702 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.208262 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.208971 25105 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.209702 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.144083 31641 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.179168 31640 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.202538 25022 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:51.197121 31770 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:51.204885 24848 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 8 to 12, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 12 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.190569 31461 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.145844 31394 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.158243 31751 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20240426 13:21:51.239312 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.225083 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.232160 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.248510 25026 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.249652 25099 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20240426 13:21:51.227890 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.251873 24956 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:51.252808 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.244457 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:51.254539 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 338575us
I20240426 13:21:51.254667 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.255600 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.252403 24956 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:51.256412 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.187251 31776 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20240426 13:21:51.262348 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.263479 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.264168 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.264575 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.264993 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:51.265969 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.270105 24848 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 6 to 12, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 12 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.271759 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.227687 31770 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:51.268337 25023 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:51.241081 31635 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.269261 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:51.269615 31640 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.269991 31361 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:51.268095 31736 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.169183 31658 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.268286 31630 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.274741 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.302817 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:51.303099 25030 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.277069 25102 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:51.303802 25102 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.239545 31580 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:51.220558 31292 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.273655 24956 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.283083 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.225517 31583 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.274585 31575 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:21:51.330816 24848 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 12, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 12 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.308228 30665 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.333072 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.332785 25102 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.333669 25099 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:51.333832 30665 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.334352 25102 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.334568 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.340035 24949 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:51.332827 24848 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 14, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.343147 25102 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:51.276300 31325 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.281250 31770 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.323746 25026 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:51.352214 25102 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20240426 13:21:51.288625 31706 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:51.287000 31361 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:51.356432 24954 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:51.358309 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.266084 31650 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
W20240426 13:21:51.365377 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62651.9us
I20240426 13:21:51.379221 24950 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20240426 13:21:51.379108 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.384406 25100 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
I20240426 13:21:51.316860 31639 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:51.389052 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 333423us
I20240426 13:21:51.392053 31713 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:21:51.393347 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110357us
I20240426 13:21:51.396216 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.393875 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:51.364753 31575 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.379036 31361 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:51.388067 31788 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20240426 13:21:51.307996 31736 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:51.380896 31635 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.390025 31290 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:51.388490 31753 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240426 13:21:51.378059 31580 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.353286 31630 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.422963 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:51.430011 31631 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18.
I20240426 13:21:51.415138 31800 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:51.431205 31800 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:21:51.432297 31790 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20240426 13:21:51.432224 31635 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.434315 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:51.434516 31635 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.434913 25096 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:51.438760 31461 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.440172 31713 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.440636 31713 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.440941 31626 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:51.441468 31626 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.443368 31626 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.402443 31568 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.446460 31658 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:51.447530 31568 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.448146 25103 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:51.453078 31580 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.455410 31630 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.411181 31744 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.456342 31744 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.457284 24956 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:51.458814 31744 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.411832 31751 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:21:51.460673 24848 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 5 to 11, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 11 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:21:51.460721 31557 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:51.463246 31721 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:51.465278 31227 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.004s
I20240426 13:21:51.419292 31641 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.467443 31583 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.467279 31643 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.471462 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:21:51.414247 31290 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:51.445063 31575 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.473132 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.484547 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.488142 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:51.480635 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.471612 25030 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:51.500334 25098 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:51.504112 25029 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:51.530292 25103 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:21:51.531682 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.532655 25103 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.539094 24996 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
I20240426 13:21:51.539562 24996 leader_election.cc:401] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:51.542575 25103 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
W20240426 13:21:51.543190 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 288165us
I20240426 13:21:51.481873 31290 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:51.545562 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.548877 25026 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:21:51.498584 31345 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:51.498994 31641 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:51.554934 25103 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:51.563879 31736 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.566444 31774 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:51.566954 31774 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:51.567863 31774 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.569402 31774 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.574339 31633 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:21:51.574854 31633 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.575836 31633 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.579231 31707 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:51.580013 31633 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:51.581461 31811 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:21:51.583303 31811 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:51.583634 31811 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.584089 31769 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:21:51.553758 31643 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.585631 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.585659 31811 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.587116 31721 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.587303 31811 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.588284 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:51.588848 25615 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:51.589154 31292 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.586623 25101 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.496840 31631 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.055s
I20240426 13:21:51.552683 31227 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20240426 13:21:51.432260 31800 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.608530 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:51.596758 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 331802us
I20240426 13:21:51.597206 24956 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:21:51.610188 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:51.610563 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.611172 24956 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:21:51.598944 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.600303 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.600504 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:51.613184 25624 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.539069 31626 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20240426 13:21:51.616278 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.590615 31706 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:21:51.627660 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.627687 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:51.617712 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.619690 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.629382 24949 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:51.629338 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.629894 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.630744 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:51.610065 31662 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.650s	user 0.003s	sys 0.006s
I20240426 13:21:51.634531 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.622005 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.625500 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.636230 24947 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:51.616500 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.614488 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.636826 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.637207 24953 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:51.617022 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.636003 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.633123 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.602299 31641 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.630144 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:51.636361 25100 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:51.648573 25619 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:51.651990 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.681591 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.651835 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.651541 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.650799 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.685108 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:51.696316 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.696985 24955 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:51.699662 31299 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20240426 13:21:51.701857 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:51.702243 31714 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.000s
I20240426 13:21:51.702654 31714 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.703905 25617 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:51.691620 31746 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20240426 13:21:51.705206 31714 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.706171 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.635466 31631 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:51.647408 31568 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:51.642769 31580 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.652359 31747 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:51.610518 31713 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.719918 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:51.693570 31461 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:51.727237 31580 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.738577 31580 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.740324 31713 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.719333 31631 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:51.748509 31641 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.753577 31766 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:21:51.754292 31766 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.756640 31630 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:51.757040 31630 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:51.757860 31630 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:51.758879 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 118373us
I20240426 13:21:51.760036 31575 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:51.760421 31575 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:51.760496 31702 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.001s
W20240426 13:21:51.761448 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 79936.9us
I20240426 13:21:51.761123 31702 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.762948 31804 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:51.763551 31804 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.764243 31325 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:51.764382 31804 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.765657 31568 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.766525 31568 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.768316 24950 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:51.768347 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.769580 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.777637 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.643695 31361 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.782761 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:51.787410 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:51.790823 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 89028.5us
W20240426 13:21:51.793877 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 257220us
I20240426 13:21:51.552495 24848 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 9 to 14, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:21:51.815277 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.755422 31766 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.788602 31580 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:51.822302 30665 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:51.828351 30665 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:51.643819 31662 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.730957 31770 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.881721 31770 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.882544 25022 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21.
I20240426 13:21:51.773689 31461 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.810856 31798 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.888429 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.886968 25624 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:51.889225 25025 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:51.811136 31325 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:21:51.843312 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.842368 31290 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.842672 31807 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.847993 31361 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:51.818486 31292 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20240426 13:21:51.848073 31653 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20240426 13:21:51.848964 31812 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20240426 13:21:51.851135 24949 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:51.761181 31575 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.862800 31706 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20.
I20240426 13:21:51.864760 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.865108 31715 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20240426 13:21:51.906533 24848 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 11, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 11 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:21:51.914716 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:51.915408 25024 raft_consensus.cc:2354] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:51.865761 25104 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:21:51.866523 31702 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.866608 31557 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:51.919935 31557 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.921902 31557 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.886555 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.887468 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:51.873221 31714 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:51.890444 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47191.6us
I20240426 13:21:51.926528 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.891178 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.898618 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:51.898185 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:51.900050 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.906466 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.909642 31630 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:51.909557 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.910893 25029 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20240426 13:21:51.911064 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.928782 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.911942 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.913062 31575 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:51.899200 31721 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:51.914256 25031 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23. Ops: []
I20240426 13:21:51.840255 30665 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:51.931241 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43848.5us
I20240426 13:21:51.915328 31290 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.870146 24952 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:51.895615 31800 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:51.917502 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:51.923326 24953 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:51.939199 25027 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20240426 13:21:51.938421 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:51.925170 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:51.941187 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:51.943331 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.833058 31809 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.083s
I20240426 13:21:51.925768 25105 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:51.946806 31818 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:51.947305 31818 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.949321 31818 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.950572 31361 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:51.951009 31361 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.952322 31800 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:21:51.952939 31800 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.953351 31800 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.955178 31800 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.955673 24951 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:51.904163 31807 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:51.895637 31798 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.944847 31580 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:51.937433 31714 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.926458 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:51.929064 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:51.956009 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:51.956478 24954 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:51.965698 25028 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:51.940181 24956 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:51.944310 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:51.977106 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.994752 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:51.957144 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.986548 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:51.975875 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:51.995534 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:51.956414 31702 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:51.972620 31809 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:21:51.997613 31809 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:51.998265 31776 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:51.998687 31776 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:51.998351 31809 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:51.999377 31290 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:52.000066 25025 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:52.000133 31557 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:21:51.997105 31643 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:51.972460 31715 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.001490 31557 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:52.001791 31643 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.996939 25026 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:21:52.001950 31715 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.002679 31643 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.003190 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.978135 31580 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:51.957747 31514 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.047s
I20240426 13:21:51.971859 31583 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:51.997007 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:51.965895 31807 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:51.969303 31798 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.978561 31821 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:51.973654 31714 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:51.972874 31706 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:21:52.003988 31715 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.004349 31721 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:52.009114 31721 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.009543 31721 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:52.011376 25622 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:51.999383 31776 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.004997 31580 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.006480 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.006542 31514 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:52.006944 31821 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.014211 31514 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:52.014394 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.005894 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:52.015302 25617 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.015552 31821 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:51.972730 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 321247us
I20240426 13:21:52.016321 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.017086 31751 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:52.017664 31751 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:52.018110 31746 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:52.019254 31746 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:21:52.016813 31361 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.022146 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:52.023074 31653 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.023186 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.023411 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.023775 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:52.023658 31653 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.025638 31653 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.007851 31706 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:51.967519 31461 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:52.028437 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:51.999876 31809 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:52.030055 25102 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:21:52.030755 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:52.031248 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.032541 31662 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:52.033195 24921 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21.
I20240426 13:21:52.033568 24921 leader_election.cc:401] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:21:52.034118 31753 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.038259 31818 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.038930 31818 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.039096 31557 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.041370 31818 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.041997 25027 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20240426 13:21:52.042147 31299 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.053774 31299 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.018265 31751 consensus_queue.cc:1225] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:21:52.042769 31715 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.044872 31821 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:52.061652 31821 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:52.013193 31776 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:52.062114 31821 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:52.028606 31706 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:21:52.048169 25617 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:52.048434 31799 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:52.060446 31715 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.072582 24947 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 17
W20240426 13:21:51.989888 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51526us
I20240426 13:21:52.042723 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.991201 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
W20240426 13:21:51.991117 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47838.2us
I20240426 13:21:52.063235 31821 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.078191 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.078601 31821 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:52.079267 31818 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:52.079782 31736 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:52.079895 31818 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:52.080271 31736 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.080932 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.081616 25096 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:21:52.081159 31818 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.083750 31715 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.078225 31751 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:52.084937 31751 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.081179 31736 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.086979 31751 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.055872 31299 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.087806 24949 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:52.090489 24950 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:52.091756 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:51.989746 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:21:52.097121 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 107495us
I20240426 13:21:52.098345 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.100735 24949 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:52.101881 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.102595 24956 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:52.104137 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.105343 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:52.106436 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.107036 24956 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:52.108283 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.108595 24947 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:52.109567 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.111375 24953 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:52.113332 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.113826 31568 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:52.113195 31805 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20240426 13:21:52.114317 31583 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:52.114451 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.114842 31583 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.115139 25104 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:52.115315 31583 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:52.116228 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.116828 25100 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:52.117048 31227 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Stepping down as leader of term 22
W20240426 13:21:52.117513 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83117.1us
I20240426 13:21:52.117758 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.118384 24921 leader_election.cc:398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:52.117504 31227 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.113734 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.119693 31227 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.120730 31227 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.118685 24921 leader_election.cc:401] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20240426 13:21:52.118655 25101 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:52.122567 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:52.124364 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 81694.2us
I20240426 13:21:52.130626 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:52.131199 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.132063 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.132932 24949 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:52.134016 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.134819 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.087278 31736 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.144666 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.146828 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.146829 24949 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:21:52.148383 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:52.109768 31299 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:52.155390 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.156239 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.156775 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.157352 25030 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:52.157620 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.154207 25104 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20240426 13:21:52.154072 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:52.153228 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.153327 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.159615 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:52.160759 31753 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21.
W20240426 13:21:52.161329 31830 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20240426 13:21:52.161571 31429 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:52.162027 31429 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.162402 31429 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:52.122229 31798 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:52.169095 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.170197 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.145905 31514 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:52.136595 31809 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:52.151832 31746 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:21:52.174290 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.146168 31461 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:52.151677 31653 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:52.130370 31568 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:52.188396 25104 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:52.189024 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.184890 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.185134 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:52.184921 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54844.7us
I20240426 13:21:52.169731 31325 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:52.165226 31706 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.153779 31805 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20240426 13:21:52.174583 31702 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:52.173238 31798 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:52.199087 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.175127 31715 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:52.172240 31361 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:52.200688 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.176321 31821 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:52.208384 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:52.176393 31299 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.211076 24956 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20240426 13:21:52.184342 31461 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:52.213757 25030 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:52.217615 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:52.214546 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56727.3us
I20240426 13:21:52.207350 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.225142 24954 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:52.215461 31821 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.226303 31821 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.226825 31299 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.184514 31653 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:52.275283 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57776.2us
W20240426 13:21:52.272625 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 114435us
I20240426 13:21:52.208956 31715 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.300278 31715 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:52.305447 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.307111 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.313318 31429 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.314615 31821 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.316017 25025 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:21:52.316180 25624 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:52.210356 31580 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:52.317943 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:21:52.328577 31580 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.318336 31568 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.329674 31807 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:52.330744 31580 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.209355 31798 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:52.331866 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.279312 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:52.277990 31746 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:52.282976 24952 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:21:52.230687 31461 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:52.284816 31635 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:21:52.287300 25617 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20240426 13:21:52.287866 31803 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:52.288789 31828 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20240426 13:21:52.245203 31653 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:52.290709 31583 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.206089 31721 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.291440 24950 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20240426 13:21:52.295910 31806 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:52.182344 31799 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20240426 13:21:52.206393 31641 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:52.318272 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:52.319680 31706 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.321508 24949 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20240426 13:21:52.355566 24949 pending_rounds.cc:76] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:21:52.324893 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.325618 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:52.327253 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:52.334086 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54886.8us
I20240426 13:21:52.241235 31292 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:21:52.239387 31325 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:52.341657 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:52.328178 31824 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20240426 13:21:52.346851 25096 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20240426 13:21:52.336259 31361 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:52.364787 31806 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:52.370057 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.370496 31799 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:52.370685 24954 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.370271 31806 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.370932 31799 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.371748 31799 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.372220 31292 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:52.372975 31799 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:52.373284 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.374061 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.374861 24950 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:52.365206 31841 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:52.376048 31841 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.376178 25103 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:52.376436 31841 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.324223 25105 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: [17.9-19.10]
I20240426 13:21:52.355942 24949 pending_rounds.cc:97] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20240426 13:21:52.381235 24949 raft_consensus.cc:2866] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:52.365370 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:52.356446 25100 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:52.384753 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.382287 24954 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:21:52.384789 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:52.387964 25624 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:52.366602 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.316280 31429 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.390718 31840 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.339337 31746 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.387329 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.392683 31753 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:52.373145 31292 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.393703 31840 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.398432 31721 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:52.398643 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.399032 31721 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:52.399472 25617 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:52.400822 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.402285 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.402720 31394 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:52.403244 25027 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:52.399583 31721 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.406093 31721 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.320591 31643 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
W20240426 13:21:52.409226 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47002.3us
I20240426 13:21:52.410843 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.411420 24954 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:21:52.413240 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:52.395800 31840 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:52.426584 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61296.9us
I20240426 13:21:52.427448 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.427816 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:52.427662 25025 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 15. Current term is 16. Ops: []
I20240426 13:21:52.373150 31583 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.355203 31641 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.438359 31641 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.434851 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.435200 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.435482 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.436121 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.347491 31811 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
I20240426 13:21:52.437856 31706 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:21:52.437683 31583 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:21:52.392830 31746 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.454121 25028 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:21:52.444576 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:52.454634 25624 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:52.455045 25022 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:21:52.458925 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:52.459843 31799 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:52.460168 31575 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.426340 31766 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20240426 13:21:52.460767 31575 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.460261 31799 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.461938 31799 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.462813 31575 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.460196 31706 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:52.465066 31706 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.471591 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.472400 25615 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:52.453001 31841 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.473562 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.475171 31841 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.475481 31823 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:52.475870 31823 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.476195 31823 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:52.477447 31799 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.430430 31514 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:21:52.478842 31514 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:52.479419 31799 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.479528 31361 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:21:52.482013 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:52.483214 31227 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:21:52.484571 31461 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.485122 31227 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:21:52.485688 31227 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.487001 31461 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:52.486586 31227 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.488873 31325 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.490916 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.492647 31635 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.493136 31635 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.353508 31803 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.495867 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.497215 31803 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.497274 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.497603 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:52.497833 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:52.498031 25024 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.498456 25028 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 25.
I20240426 13:21:52.499523 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.499612 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:52.497386 25100 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.498159 25615 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:52.505496 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:52.509238 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64772.8us
I20240426 13:21:52.509837 24947 raft_consensus.cc:2354] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:52.512804 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.517575 25025 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:52.518628 25105 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:21:52.519472 31798 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.453132 31821 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.525539 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.526115 25103 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.520500 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.522048 25096 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:21:52.514106 31766 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:52.532780 31803 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:21:52.533190 31803 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:52.533511 31803 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:21:52.535454 25103 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:21:52.538136 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:52.539788 25617 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:21:52.540971 24949 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23.
I20240426 13:21:52.541970 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.515555 24954 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:52.479306 31514 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:52.553408 25588 leader_election.cc:398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:52.559753 25588 leader_election.cc:401] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20240426 13:21:52.559926 24947 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:21:52.556015 25069 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23.
I20240426 13:21:52.560451 25069 leader_election.cc:401] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20240426 13:21:52.436055 31290 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:52.556703 24952 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:52.557139 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:52.456573 31769 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:52.547955 24954 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:21:52.580899 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.554199 31721 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:52.586261 31721 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:52.589774 31721 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:52.597077 31583 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23.
I20240426 13:21:52.598701 24953 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: []
I20240426 13:21:52.586338 31635 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.599642 31361 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:52.600487 31429 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:21:52.600541 31361 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.601430 31429 consensus_queue.cc:1225] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20240426 13:21:52.601982 31429 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:21:52.602283 31429 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:52.602667 31361 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.560114 31227 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:21:52.619313 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:52.635324 25615 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:52.645542 31653 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.646890 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:52.637128 31715 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.649566 31715 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.645201 25105 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 14.
I20240426 13:21:52.652434 24951 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:52.651321 31721 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.654408 31721 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.657454 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:21:52.572921 31290 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.659770 25030 raft_consensus.cc:2354] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:52.528622 31641 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.670892 25100 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
W20240426 13:21:52.682363 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63202.3us
I20240426 13:21:52.795193 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:52.572651 31841 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20240426 13:21:52.655318 31461 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20240426 13:21:52.670689 31751 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:52.671669 31798 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:52.698014 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:21:52.655282 31557 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:52.712502 31807 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:52.723018 31811 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:52.672959 31429 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.716501 31635 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:52.746592 31803 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.749122 31653 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.739814 31823 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.752779 31514 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.755757 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.757884 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:52.791929 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.014s	sys 0.027s
W20240426 13:21:52.828685 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.015s	sys 0.027s
I20240426 13:21:52.828143 25103 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:52.799100 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:52.801652 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:52.798452 31290 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:52.801425 31580 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:21:52.832310 25617 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:52.833509 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:52.837886 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 325183us
I20240426 13:21:52.816707 31751 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.812407 31394 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20240426 13:21:52.851331 24956 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:52.858016 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:52.859679 24954 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:21:52.860996 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:52.861696 24953 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:52.821135 31807 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:52.832149 31568 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.023s	sys 0.052s
I20240426 13:21:52.863723 25615 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:52.865522 25615 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:52.836288 31630 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:21:52.859170 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:52.837550 31798 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:52.870905 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.839419 31805 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:21:52.819155 31811 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:52.888837 25624 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:21:52.878089 25027 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:21:52.823246 31635 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.897485 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:52.871881 31568 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.023s	sys 0.052s
I20240426 13:21:52.945914 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:52.943622 25024 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:21:52.965404 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66498.9us
I20240426 13:21:52.974397 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:52.853435 31643 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.093s
I20240426 13:21:52.984179 25623 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:21:52.865882 31580 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.097s
I20240426 13:21:52.872474 25615 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:52.896564 31630 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:52.961269 25030 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20240426 13:21:52.995699 25030 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 11
I20240426 13:21:52.995947 25030 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20240426 13:21:52.996202 25030 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:52.994275 25615 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:52.955703 31805 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:53.028579 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:53.028851 25615 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20240426 13:21:53.029812 25615 pending_rounds.cc:76] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 15
I20240426 13:21:53.033049 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:53.041822 25615 pending_rounds.cc:97] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20240426 13:21:53.042542 25615 raft_consensus.cc:2866] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:53.012296 31568 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.063097 24921 leader_election.cc:398] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:53.057837 31840 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:53.064775 31630 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.066552 31803 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:52.991633 31292 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.063s
I20240426 13:21:52.972144 31653 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20240426 13:21:53.073943 25028 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20240426 13:21:53.063555 24921 leader_election.cc:401] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:21:53.072409 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 231217us
I20240426 13:21:53.075552 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:53.081382 31852 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20240426 13:21:53.083770 31855 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20240426 13:21:53.084335 31514 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:53.084573 31848 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20240426 13:21:53.085835 31653 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:21:53.074486 25028 pending_rounds.cc:76] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 13
I20240426 13:21:53.190577 25028 pending_rounds.cc:97] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20240426 13:21:53.190944 25028 raft_consensus.cc:2866] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:21:53.089326 31850 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:21:53.053159 31641 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:53.187368 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 108595us
W20240426 13:21:53.132123 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 157154us
I20240426 13:21:53.223130 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:53.234452 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:52.984488 31751 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.241068 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:53.240559 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 295193us
W20240426 13:21:53.242151 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 279730us
I20240426 13:21:53.185681 31809 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.265189 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:53.269001 25101 raft_consensus.cc:2354] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:53.269801 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:53.270198 25098 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
W20240426 13:21:53.277864 31854 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
W20240426 13:21:53.279785 31864 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20240426 13:21:53.236693 31720 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:21:53.234629 31580 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.303234 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:53.306509 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:53.259174 31823 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:53.310034 25029 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
W20240426 13:21:53.287734 31839 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:21:53.070875 31807 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.311703 25024 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:21:53.325755 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:53.327562 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:53.328270 25028 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:21:53.236263 31798 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20240426 13:21:53.332680 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:53.288537 31803 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.336954 25025 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20240426 13:21:53.349799 24951 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:21:53.332577 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:53.341192 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.353672 24954 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:53.092519 31568 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.356644 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3370) took 1022 ms. Trace:
I20240426 13:21:53.269960 31809 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.299327 31821 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:21:53.356062 25623 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:53.311939 31361 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:53.319782 31643 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:53.336143 31823 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:21:53.374361 25104 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:53.290509 31641 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.052s
I20240426 13:21:53.328317 31811 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:53.368849 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:53.358696 25026 rpcz_store.cc:276] 0426 13:21:52.333729 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:52.333835 (+   106us) service_pool.cc:225] Handling call
0426 13:21:52.342383 (+  8548us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:52.343111 (+   728us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:52.343142 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:52.347277 (+  4135us) log.cc:842] Serialized 26 byte log entry
0426 13:21:52.763812 (+416535us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:52.763865 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:52.763955 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:53.356424 (+592469us) raft_consensus.cc:1671] finished
0426 13:21:53.356444 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:53.356570 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7494,"raft.queue_time_us":888,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":7210,"thread_start_us":376232,"threads_started":1,"wal-append.queue_time_us":539654}
I20240426 13:21:53.369998 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:53.358307 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 120081us
I20240426 13:21:53.308106 31325 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20240426 13:21:53.342845 31635 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:53.352185 31803 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.375784 25098 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20240426 13:21:53.312569 31580 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.349030 31869 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
W20240426 13:21:53.381731 31853 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20240426 13:21:53.357143 31394 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20240426 13:21:53.384853 31859 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:53.393083 25022 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:53.270948 31805 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:53.374034 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56755.5us
I20240426 13:21:53.393513 31879 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:53.400285 25101 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:53.399382 31872 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.378983 31643 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.420377 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:53.459513 25103 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:53.433748 24949 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20240426 13:21:53.460356 24949 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 9
I20240426 13:21:53.460683 24949 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20240426 13:21:53.460964 24949 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:53.435256 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.437220 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:53.406651 31823 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:53.406111 31865 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20240426 13:21:53.443075 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 202021us
I20240426 13:21:53.468964 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:53.444926 31881 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:53.445572 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 177064us
W20240426 13:21:53.406154 31856 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
W20240426 13:21:53.415766 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102170us
I20240426 13:21:53.454697 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:53.429538 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:53.487387 24954 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:53.462136 24953 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:21:53.477319 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:53.372710 31361 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.492887 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:53.493538 25105 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:53.493963 25105 raft_consensus.cc:603] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:21:53.490092 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52985.2us
I20240426 13:21:53.496809 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:53.498386 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 43786.6us
I20240426 13:21:53.497536 25026 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:53.504765 25024 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:53.507001 25027 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20240426 13:21:53.507333 25027 pending_rounds.cc:76] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 9
I20240426 13:21:53.507525 25027 pending_rounds.cc:97] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20240426 13:21:53.507759 25027 raft_consensus.cc:2866] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:53.499802 24955 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:53.521502 24954 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:53.484349 31869 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:53.533723 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:53.486498 31867 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:53.537684 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:53.539891 25027 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:53.544548 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3368) took 1226 ms. Trace:
I20240426 13:21:53.546895 31823 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:53.548055 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70833.5us
I20240426 13:21:53.548787 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:53.517994 31568 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:53.522203 31557 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:21:53.544982 25023 rpcz_store.cc:276] 0426 13:21:52.318223 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:52.337361 (+ 19138us) service_pool.cc:225] Handling call
0426 13:21:52.344361 (+  7000us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:52.344685 (+   324us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:52.344728 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:52.346195 (+  1467us) log.cc:842] Serialized 26 byte log entry
0426 13:21:52.478261 (+132066us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b48000b19d8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:21:52.792359 (+314098us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:52.792398 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:52.792491 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:53.537987 (+745496us) raft_consensus.cc:1671] finished
0426 13:21:53.538008 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:53.544484 (+  6476us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1605,"raft.run_cpu_time_us":6992,"raft.run_wall_time_us":126915,"spinlock_wait_cycles":4077696,"thread_start_us":405496,"threads_started":1,"wal-append.queue_time_us":647166}
I20240426 13:21:53.530774 31770 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:53.555790 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74024.3us
I20240426 13:21:53.537690 31809 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:53.563094 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57461.8us
I20240426 13:21:53.547662 31886 raft_consensus.cc:2741] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
W20240426 13:21:53.576203 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65102.5us
I20240426 13:21:53.481283 31872 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.557960 31643 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:53.581804 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:53.578763 24921 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17.
I20240426 13:21:53.587469 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:53.488741 31881 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.602888 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:53.604722 24954 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:53.607312 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.609931 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.602988 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.629559 24947 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:53.632298 24955 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:53.636267 25617 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:53.663774 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66913us
W20240426 13:21:53.666234 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69092.7us
I20240426 13:21:53.669409 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:53.669756 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:53.670572 25617 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
W20240426 13:21:53.671859 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 162996us
W20240426 13:21:53.674027 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81360.8us
I20240426 13:21:53.565510 31568 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20240426 13:21:53.590752 31798 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:53.677171 25098 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20240426 13:21:53.681313 24954 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20240426 13:21:53.691211 24955 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:21:53.668499 31751 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:53.596504 31643 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.698112 25589 leader_election.cc:398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21.
I20240426 13:21:53.582710 31809 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.097s
I20240426 13:21:53.699635 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:53.578755 31887 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.133s
I20240426 13:21:53.697211 31751 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:53.672318 31766 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:21:53.702471 31720 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.704391 31461 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.707829 31803 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:53.751516 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:53.708631 31580 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:53.713382 31641 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:53.720837 31290 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.866s	user 0.010s	sys 0.001s
W20240426 13:21:53.752974 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 153330us
I20240426 13:21:53.758401 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:53.757993 25028 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
I20240426 13:21:53.762154 25104 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20240426 13:21:53.747195 31751 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:53.768530 25098 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:21:53.772043 25031 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 25. Current term is 26. Ops: []
I20240426 13:21:53.708920 31643 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.753463 31870 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:53.789742 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:53.714200 31866 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.225s
I20240426 13:21:53.667410 31869 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.764587 31361 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:53.759117 31720 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.769742 31568 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:53.796756 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:53.758950 31461 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.807054 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:53.809098 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:53.790187 31870 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:53.768611 31290 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.768273 31641 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.814623 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:53.817934 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:53.803926 31653 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:53.813689 31429 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:53.816974 31870 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:53.836463 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:53.837435 25096 raft_consensus.cc:2396] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20240426 13:21:53.829993 31849 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.741s	user 0.005s	sys 0.001s
I20240426 13:21:53.830477 31290 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:53.834050 25025 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:53.806821 31568 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.843027 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:53.844831 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:53.848860 25098 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20240426 13:21:53.848757 31899 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20240426 13:21:53.849504 31860 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:21:53.849933 31643 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:53.824318 31889 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:21:53.794018 31888 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.228s
I20240426 13:21:53.800618 31843 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.084s
I20240426 13:21:53.850730 31908 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:53.805800 31887 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.023s
I20240426 13:21:53.825059 31902 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:53.868878 31290 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:53.815374 31809 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
W20240426 13:21:53.850631 31897 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20240426 13:21:53.820919 31872 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:53.883579 31910 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:53.884147 31910 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:53.884682 31891 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20240426 13:21:53.836340 31580 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:53.839581 31881 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:53.856274 31776 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:53.842434 31292 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:21:53.870611 31299 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:53.886289 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:53.887694 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54129.5us
I20240426 13:21:53.835819 31653 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:53.874990 31643 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:21:53.933477 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.881649 31902 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:53.936724 24951 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:53.878271 31908 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:53.937845 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:53.938869 31894 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20240426 13:21:53.892367 31580 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:53.949219 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 77762.3us
I20240426 13:21:53.950201 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:53.950964 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:53.951511 25615 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:21:53.952958 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.953879 24949 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:21:53.955264 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:53.945924 31895 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20240426 13:21:53.957556 25617 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20240426 13:21:53.959857 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 181868us
W20240426 13:21:53.945966 31907 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:21:53.942234 31902 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:53.901667 31702 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.056s
I20240426 13:21:53.949678 31580 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:53.967181 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:53.944501 31879 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.892495 31290 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:53.946848 31881 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.944084 31292 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:53.945518 31908 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:53.962560 31776 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:53.980214 24951 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:53.981564 31461 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:53.960425 31860 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:53.949012 31910 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:53.959793 31888 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:54.000315 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45176.2us
I20240426 13:21:54.002869 31881 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.003988 31583 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20240426 13:21:54.054224 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63599.9us
I20240426 13:21:54.059007 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:54.049384 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:54.066722 31915 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20240426 13:21:54.068101 31913 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:21:54.068549 31913 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:21:54.069363 31913 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.070562 31913 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:21:54.072664 24951 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:21:54.072320 31292 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.073818 31653 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:54.073841 31919 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.019s
W20240426 13:21:54.055478 31905 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20240426 13:21:54.047415 31706 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:54.079253 31920 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:54.079668 31921 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:54.079344 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:54.048303 31641 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.056557 31720 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.061196 31643 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:54.062204 31583 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:54.075584 31901 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20240426 13:21:54.086128 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.000088 31879 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.045122 31888 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.044353 31860 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.047794 31885 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.271s
W20240426 13:21:54.107200 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48241.1us
I20240426 13:21:54.193653 24948 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20240426 13:21:54.113680 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 156404us
I20240426 13:21:54.085439 31920 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:54.077826 31568 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:54.246678 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 287644us
I20240426 13:21:54.176733 31870 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.138417 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.277757 31925 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
W20240426 13:21:54.205190 31912 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20240426 13:21:54.333096 31870 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:54.333432 31920 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:54.321014 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:54.310599 31843 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:54.333727 31879 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.335068 31843 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.337134 31843 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:54.147558 31643 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.288406 31888 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:54.246228 31583 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:54.191141 31927 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:54.290514 31936 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:54.362619 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.379089 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:54.432883 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:21:54.171149 31890 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.254s	user 0.000s	sys 0.013s
W20240426 13:21:54.171907 31923 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20240426 13:21:54.442090 25617 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:21:54.176573 31911 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.243s	user 0.002s	sys 0.000s
W20240426 13:21:54.287950 31751 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.000s
W20240426 13:21:54.288627 31906 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240426 13:21:54.458087 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:54.442267 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.656224 25103 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:21:54.160868 31922 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:54.147091 31876 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:54.289813 31860 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.137157 31641 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:54.616005 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.100488 31706 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.195075 31931 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:21:54.645843 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:54.270481 31290 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.434041 31881 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.657078 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.475796 31643 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:54.515933 31843 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:54.288601 31924 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.371s
I20240426 13:21:54.295094 31806 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.349s
I20240426 13:21:54.207449 31929 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.497s
I20240426 13:21:54.174443 31908 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.615200 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:54.671741 25104 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20240426 13:21:54.632833 31751 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.099864 31921 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.676584 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.687613 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.149360 31776 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.432873 31927 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.439283 31936 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:54.711383 31904 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.957s	user 0.001s	sys 0.000s
W20240426 13:21:54.711673 31580 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.004s
I20240426 13:21:54.711854 31936 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.712152 31580 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.692574 31843 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.433665 31938 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.242s
W20240426 13:21:54.676259 31429 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.835s	user 0.007s	sys 0.003s
I20240426 13:21:54.687130 31290 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:54.734205 24952 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:21:54.665256 31922 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:54.720163 31902 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.000s
I20240426 13:21:54.738716 25618 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20240426 13:21:54.678532 31641 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.743276 25618 pending_rounds.cc:76] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 14
I20240426 13:21:54.746537 25618 pending_rounds.cc:97] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20240426 13:21:54.746819 25618 raft_consensus.cc:2866] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:21:54.735543 24948 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:54.668118 31876 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:54.816619 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.014s	sys 0.026s
W20240426 13:21:54.817025 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.014s	sys 0.026s
I20240426 13:21:54.817268 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:54.748605 31889 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:54.814108 31897 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20240426 13:21:54.836485 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.837294 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:21:54.738250 31290 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.840763 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.848582 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:54.852975 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.853768 25028 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:21:54.745802 31902 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:54.871376 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52819.7us
I20240426 13:21:54.873636 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.874001 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:54.874665 25028 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20240426 13:21:54.876117 31706 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:54.876340 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60031.7us
I20240426 13:21:54.842993 31870 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:54.839915 31947 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:54.857334 31942 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.399s	user 0.001s	sys 0.000s
I20240426 13:21:54.848177 31643 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:54.853309 31924 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:54.852819 31580 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:54.888901 31851 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.825s	user 0.007s	sys 0.001s
I20240426 13:21:54.891263 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:54.879853 31943 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
I20240426 13:21:54.896507 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:54.900993 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:54.901700 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.003s	sys 0.010s
W20240426 13:21:54.902563 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.003s	sys 0.010s
I20240426 13:21:54.876549 31843 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.893697 31633 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:54.882019 31706 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:54.838035 31889 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.898813 31936 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:54.957332 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:54.899118 31580 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:54.894490 31643 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.883138 31870 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.896402 31924 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:54.884397 31947 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:54.905542 31886 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:55.005312 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3227) took 2623 ms. Trace:
I20240426 13:21:54.979041 25028 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:55.009594 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64296.1us
W20240426 13:21:54.979660 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139444us
I20240426 13:21:54.909008 31946 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:54.918895 31843 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:55.010926 31955 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
W20240426 13:21:55.086243 31863 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.028s	user 0.015s	sys 0.000s
I20240426 13:21:54.944944 31633 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:55.045344 31920 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.711s	user 0.005s	sys 0.005s
I20240426 13:21:54.919824 31751 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.128640 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.011823 31876 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:54.979712 31936 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:55.135866 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.047164 31806 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:55.126628 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:55.112901 24950 rpcz_store.cc:276] 0426 13:21:52.382292 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:52.382425 (+   133us) service_pool.cc:225] Handling call
0426 13:21:52.390649 (+  8224us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:52.390990 (+   341us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:52.391021 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:52.391428 (+   407us) log.cc:842] Serialized 26 byte log entry
0426 13:21:52.864434 (+473006us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:21:52.864495 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:52.866106 (+  1611us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:54.832836 (+1966730us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:55.002978 (+170142us) raft_consensus.cc:1671] finished
0426 13:21:55.003011 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:55.005239 (+  2228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":232,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":6438,"spinlock_wait_cycles":18524544,"thread_start_us":472557,"threads_started":1,"wal-append.queue_time_us":632701}
I20240426 13:21:55.011482 31908 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:55.088831 31902 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.086719 31290 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.013312 31958 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:55.081871 31952 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:54.985718 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109581us
I20240426 13:21:55.010299 31641 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:55.118028 31886 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:55.005509 31927 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.090530 31921 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:55.150584 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
W20240426 13:21:55.085280 31963 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20240426 13:21:55.151623 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 268993us
I20240426 13:21:54.997079 31881 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:55.153117 31936 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:21:55.255532 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 129003us
I20240426 13:21:55.029284 31922 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:55.258425 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 357040us
I20240426 13:21:54.975126 31706 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:55.146440 31633 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:55.271975 31892 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:55.131572 31580 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:55.274667 31958 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:55.255220 31751 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:55.140780 31843 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.291149 31881 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.301010 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:55.281014 31886 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:55.268538 31908 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:21:55.302098 24920 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:55.319281 25623 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:21:55.133921 31946 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.153036 31920 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:55.258080 31806 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:55.292248 31641 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:55.324002 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:55.295949 31922 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:55.341995 25617 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:21:55.280589 31952 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:55.351270 25096 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:55.353399 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56033.7us
I20240426 13:21:55.360580 31557 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:55.361147 31557 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.363219 31557 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.365252 31946 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:55.366324 31643 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:55.368204 31806 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:55.351732 31908 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:55.369012 31580 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:55.369599 31908 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.339434 31988 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:55.320356 31751 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.370146 31924 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.298604 31892 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:55.383651 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 81634.4us
I20240426 13:21:55.351281 31985 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:55.522969 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 167172us
I20240426 13:21:55.534430 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:55.535369 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:55.384732 31927 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:55.388618 31841 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:55.516160 31913 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.439s	user 0.010s	sys 0.005s
I20240426 13:21:55.347818 31641 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.641311 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.352344 31922 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:55.661054 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.736219 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.737001 24949 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:55.654807 31870 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:55.511286 31988 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.733675 31947 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:55.855154 24949 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:21:55.823654 31988 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:55.887039 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.639672 31927 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:55.907897 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.655316 31841 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:55.970932 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.029s	sys 0.047s
W20240426 13:21:55.971534 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.029s	sys 0.047s
W20240426 13:21:55.971937 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 662334us
I20240426 13:21:55.826413 31870 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:55.983100 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.000s	sys 0.009s
I20240426 13:21:55.985531 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:55.632201 31985 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.023320 31971 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:55.977068 31871 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.000s	sys 0.016s
I20240426 13:21:56.041060 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:55.977973 31706 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.028535 31859 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:56.024124 31881 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:55.929770 31843 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:56.030789 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 495423us
I20240426 13:21:56.198773 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:56.068289 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.000s	sys 0.010s
I20240426 13:21:56.072309 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:56.072386 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.004s	sys 0.004s
W20240426 13:21:56.146155 25622 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.005s	sys 0.005s
I20240426 13:21:56.141088 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
W20240426 13:21:56.200435 25622 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.006s	sys 0.005s
W20240426 13:21:56.200362 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.005s	sys 0.004s
W20240426 13:21:56.201483 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 913534us
W20240426 13:21:56.072747 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.000s	sys 0.009s
W20240426 13:21:56.202170 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.000s	sys 0.009s
W20240426 13:21:56.202507 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 915011us
I20240426 13:21:56.203248 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:56.179410 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.621s	user 0.023s	sys 0.029s
I20240426 13:21:56.032647 31913 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.038444 31958 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.221024 31841 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.221561 31958 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.222074 31958 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.222704 31984 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:56.223238 31984 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:56.223655 31984 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:21:56.182395 24954 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.002s	sys 0.010s
I20240426 13:21:56.184599 31999 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20240426 13:21:56.239252 25615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.014s	sys 0.017s
I20240426 13:21:56.186631 31641 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.180779 31971 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.189810 31998 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:56.190464 31991 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.031788 31978 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:56.238065 31583 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.801s	user 0.012s	sys 0.000s
W20240426 13:21:56.211773 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.653s	user 0.023s	sys 0.029s
W20240426 13:21:56.183300 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.005s	sys 0.008s
I20240426 13:21:56.199957 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.215175 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:56.219558 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:56.149160 31871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.000s	sys 0.017s
I20240426 13:21:56.253752 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:21:56.228717 31751 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.221042 31994 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:56.234794 24954 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.002s	sys 0.010s
I20240426 13:21:56.237193 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:56.257612 25623 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:56.258809 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2784) took 2569 ms. Trace:
I20240426 13:21:56.062806 31983 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:56.239799 25615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.014s	sys 0.017s
I20240426 13:21:56.032370 31870 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.182328 31859 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:56.063540 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:56.251503 25619 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.010s	sys 0.028s
I20240426 13:21:56.229705 31958 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.233543 31843 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.268693 31843 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.269886 31870 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.270364 31859 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:56.270712 31843 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.260128 31924 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.272259 31924 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.273399 31892 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.274637 31927 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.275197 31927 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:56.275509 31983 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.276873 31922 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.277541 31983 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:56.277797 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74546.9us
W20240426 13:21:56.263727 25619 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.010s	sys 0.029s
W20240426 13:21:56.253477 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 2.69507e+06us
W20240426 13:21:56.254089 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.967s	user 0.005s	sys 0.009s
W20240426 13:21:56.285435 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 998398us
I20240426 13:21:56.081322 31920 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.260511 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.261205 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:56.263413 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 199986us
I20240426 13:21:56.254240 24949 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:56.294584 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:56.229564 31913 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.321593 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:56.331604 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.259205 24954 rpcz_store.cc:276] 0426 13:21:53.689603 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.689753 (+   150us) service_pool.cc:225] Handling call
0426 13:21:53.690081 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:53.690337 (+   256us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:53.690696 (+   359us) log.cc:842] Serialized 14 byte log entry
0426 13:21:55.003660 (+1312964us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b1800367fc0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:21:56.257966 (+1254306us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:56.258027 (+    61us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:56.258080 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:56.258630 (+   550us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:56.258754 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40545,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2303,"raft.queue_time_us":719221,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":10696,"spinlock_wait_cycles":143862272,"thread_start_us":1235741,"threads_started":2,"wal-append.queue_time_us":2342130}
I20240426 13:21:56.308514 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:56.307585 25623 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:56.346465 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.347098 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.346529 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.332382 25098 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:21:56.351908 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.352901 25098 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:56.275635 31927 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.278110 31983 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:21:56.369221 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.347703 25100 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:21:56.380470 31927 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.384721 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.385915 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.391636 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.422045 25617 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:56.277998 31922 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:56.424947 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 103420us
I20240426 13:21:56.419592 25097 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:21:56.330884 31958 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:56.303210 31992 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.430428 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.260831 31583 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.446942 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:56.438406 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.009s	sys 0.000s
W20240426 13:21:56.430440 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109976us
W20240426 13:21:56.449004 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.009s	sys 0.000s
I20240426 13:21:56.346504 31892 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.427594 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.433266 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:56.449506 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.454013 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.418613 31744 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:56.442045 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.000s	sys 0.011s
I20240426 13:21:56.454635 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:56.426506 31983 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.458447 24949 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:56.497403 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.176s	user 0.000s	sys 0.037s
I20240426 13:21:56.465998 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:56.425575 32008 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:56.503458 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.250484 31998 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.252821 31991 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.504946 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:56.439141 32004 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.523020 25617 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
W20240426 13:21:56.454932 31806 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.052s	sys 0.082s
W20240426 13:21:56.453779 31886 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.109s	user 0.002s	sys 0.006s
W20240426 13:21:56.494498 25069 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.561s (SENT) [suppressed 1 similar messages]
I20240426 13:21:56.539748 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:56.528730 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.508976 31983 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:21:56.257509 31978 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.492290 31871 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:56.452446 31969 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.009s	sys 0.000s
I20240426 13:21:56.490231 31985 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.454999 31999 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:56.532411 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.458139 31958 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.088s
I20240426 13:21:56.525156 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:56.509424 31870 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:56.439891 31922 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.287377 31994 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.517474 31744 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:56.598093 31969 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.266s	user 0.010s	sys 0.010s
W20240426 13:21:56.550135 31806 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.053s	sys 0.083s
I20240426 13:21:56.597328 31927 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:56.658501 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 192598us
I20240426 13:21:56.617755 31985 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:21:56.598866 31971 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.689507 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:56.540225 31990 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.494804 31892 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.447333 31992 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:56.674259 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 199539us
I20240426 13:21:56.671609 31902 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:56.563655 31886 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:56.551909 31936 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.255s	user 0.008s	sys 0.002s
I20240426 13:21:56.704268 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3385) took 3176 ms. Trace:
W20240426 13:21:56.724856 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58526.6us
I20240426 13:21:56.655313 32033 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.665472 31969 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:56.534162 32008 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.605981 24949 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:21:56.674640 31908 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.636963 31870 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.724006 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 976) took 1389 ms. Trace:
I20240426 13:21:56.725026 25022 rpcz_store.cc:276] 0426 13:21:53.509775 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.509955 (+   180us) service_pool.cc:225] Handling call
0426 13:21:53.534800 (+ 24845us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:53.535115 (+   315us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:53.535151 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:53.535566 (+   415us) log.cc:842] Serialized 26 byte log entry
0426 13:21:56.664993 (+3129427us) spinlock_profiling.cc:243] Waited 4.47 ms on lock 0x7b18000fb680. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291
0426 13:21:56.665138 (+   145us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:56.665174 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:56.665253 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:56.665307 (+    54us) raft_consensus.cc:1671] finished
0426 13:21:56.665319 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:56.685813 (+ 20494us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8511,"raft.queue_time_us":31081,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":10620,"spinlock_wait_cycles":10292480,"thread_start_us":43516,"threads_started":1,"wal-append.queue_time_us":331981,"wal-append.run_cpu_time_us":8938,"wal-append.run_wall_time_us":1453271}
I20240426 13:21:56.715085 31913 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.743772 24953 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20240426 13:21:56.709892 32014 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:56.552199 32004 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.715368 31990 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.751921 31658 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:56.752471 32044 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:21:56.752619 31633 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.461s	user 0.009s	sys 0.001s
I20240426 13:21:56.717315 31975 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:56.741511 31583 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.724669 31290 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:56.754437 32021 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:21:56.756237 31992 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:56.756435 32011 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.314s	user 0.000s	sys 0.000s
W20240426 13:21:56.729131 31630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.956s	user 0.031s	sys 0.031s
I20240426 13:21:56.618192 31999 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.674283 31927 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.631270 31871 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.738927 32033 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.736601 31969 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:21:56.777772 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51887.6us
I20240426 13:21:56.791432 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:56.743297 25619 rpcz_store.cc:276] 0426 13:21:55.324564 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.325302 (+   738us) service_pool.cc:225] Handling call
0426 13:21:55.343360 (+ 18058us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:55.343579 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 16
0426 13:21:56.287007 (+943428us) log.cc:842] Serialized 14 byte log entry
0426 13:21:56.713205 (+426198us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:56.713273 (+    68us) raft_consensus.cc:1631] Marking committed up to 16
0426 13:21:56.713334 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:56.713395 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:56.713493 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45942,"prepare.queue_time_us":1143251,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":119264,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":30423,"thread_start_us":1272248,"threads_started":3}
W20240426 13:21:56.855379 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 360886us
I20240426 13:21:56.901575 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.902550 25102 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:56.731276 31902 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.756781 32018 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:56.731566 31936 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.909986 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.910202 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.909883 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.781980 31975 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:56.917639 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.930187 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:56.931227 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 426353us
I20240426 13:21:56.799083 31633 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:56.910737 31630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.140s	user 0.032s	sys 0.031s
I20240426 13:21:56.961807 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:56.940356 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.962221 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.855578 31909 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:56.924098 31859 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.653s	user 0.010s	sys 0.000s
I20240426 13:21:56.941648 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:21:56.969430 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.757836 31908 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.932266 25097 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:56.970234 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:56.948257 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:56.930227 31583 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:56.962877 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:56.965507 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.918386 31811 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:56.962747 25103 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:21:56.965521 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.966249 25031 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:21:56.931931 31969 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:56.964655 31998 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.931617 32021 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:56.961175 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:56.987123 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.972987 24948 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:56.972967 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:56.937371 31992 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:21:56.978519 31870 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.979223 32037 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.005609 32037 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.006523 31908 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.954819 31975 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:57.007573 32037 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.979580 32022 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:56.982141 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.961347 31979 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:56.856077 31290 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.972779 31909 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.014362 31909 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.966017 31806 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.015020 31806 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.980248 31859 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.017194 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.017477 31859 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.017913 25022 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 16: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:56.960829 31922 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.018698 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.019423 25022 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 22: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:57.019263 31715 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:21:57.020929 31999 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.010020 32021 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.022042 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.022533 31924 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.856866 32044 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.025871 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2797) took 1363 ms. Trace:
I20240426 13:21:57.026177 32044 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.026262 24950 rpcz_store.cc:276] 0426 13:21:55.662430 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.662576 (+   146us) service_pool.cc:225] Handling call
0426 13:21:55.749999 (+ 87423us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:55.754622 (+  4623us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:55.754667 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:55.754724 (+    57us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:55.754770 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:57.025656 (+1270886us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b5400102054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:57.025702 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:57.025819 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10122624}
I20240426 13:21:57.023057 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.028930 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:57.029668 25022 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:57.029911 31866 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.030125 32041 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:21:57.030526 32041 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.031025 32041 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
W20240426 13:21:57.032637 31986 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.299s	user 0.007s	sys 0.010s
I20240426 13:21:57.033813 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2592) took 3226 ms. Trace:
W20240426 13:21:57.034819 31981 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.211s	user 0.000s	sys 0.017s
I20240426 13:21:57.035562 31886 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.034184 25023 rpcz_store.cc:276] 0426 13:21:53.807307 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.807515 (+   208us) service_pool.cc:225] Handling call
0426 13:21:53.815953 (+  8438us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:53.816327 (+   374us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:53.816354 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:53.821296 (+  4942us) log.cc:842] Serialized 26 byte log entry
0426 13:21:55.137452 (+1316156us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b18002fffe0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:21:55.152483 (+ 15031us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:21:55.152522 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:55.152587 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:57.031963 (+1879376us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:57.033584 (+  1621us) raft_consensus.cc:1671] finished
0426 13:21:57.033613 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:57.033741 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7110,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":13708,"spinlock_wait_cycles":62078208,"thread_start_us":14817,"threads_started":1,"wal-append.queue_time_us":1552108}
I20240426 13:21:57.035874 31227 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:21:57.036378 31227 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:21:57.036181 32022 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.991825 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.998071 31990 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.989419 31583 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:56.970770 31913 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.040841 32022 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.040989 31913 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.037621 31978 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:57.041754 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 3039) took 2679 ms (client timeout 1612 ms). Trace:
I20240426 13:21:57.041983 31583 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:57.042214 25096 rpcz_store.cc:269] 0426 13:21:54.362304 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.362398 (+    94us) service_pool.cc:225] Handling call
0426 13:21:57.041568 (+2679170us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:57.041682 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111843072}
I20240426 13:21:57.042865 31913 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.039650 31227 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:56.775111 32014 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.044543 31227 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:57.045974 32014 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:56.946051 32018 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.044546 31290 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.048398 32018 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.049003 32004 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.011818 31979 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:57.050724 31979 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:57.051685 31633 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.037853 25027 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:57.030382 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:57.054364 25022 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:57.001492 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.002748 31991 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.002871 32033 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.992270 31992 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:57.003716 32008 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:56.987978 31811 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.004881 31927 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.005088 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2796) took 1349 ms. Trace:
I20240426 13:21:57.069000 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:57.011459 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:57.012305 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82209.1us
I20240426 13:21:57.030468 31866 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.029026 31871 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:56.991386 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:56.993505 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.040063 31994 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.023736 32021 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.085273 25031 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20240426 13:21:57.030558 31936 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:57.069975 31984 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.846s	user 0.000s	sys 0.009s
I20240426 13:21:56.983753 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.071296 24948 rpcz_store.cc:276] 0426 13:21:55.655517 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.655667 (+   150us) service_pool.cc:225] Handling call
0426 13:21:55.731320 (+ 75653us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:57.003397 (+1272077us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:57.004871 (+  1474us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400078ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:57.005034 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2855209728}
I20240426 13:21:57.080652 31987 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:21:57.039116 32037 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.064900 31952 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:57.081144 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 108245us
I20240426 13:21:57.081235 31998 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:57.085099 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:57.036310 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3383) took 3544 ms. Trace:
I20240426 13:21:57.096542 31806 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:57.096402 25029 rpcz_store.cc:276] 0426 13:21:53.492187 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.493011 (+   824us) service_pool.cc:225] Handling call
0426 13:21:53.493758 (+   747us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:53.518593 (+ 24835us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:53.518631 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:54.857801 (+1339170us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:54.858158 (+   357us) log.cc:842] Serialized 26 byte log entry
0426 13:21:55.147294 (+289136us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:55.147328 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:55.147390 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:57.034085 (+1886695us) spinlock_profiling.cc:243] Waited 57.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:57.035871 (+  1786us) raft_consensus.cc:1671] finished
0426 13:21:57.035891 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:57.036003 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24948,"raft.queue_time_us":70649,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7264,"spinlock_wait_cycles":142187776,"thread_start_us":254713,"threads_started":1,"wal-append.queue_time_us":789441}
I20240426 13:21:57.088238 31964 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:57.086972 25026 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:57.088014 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.092080 31987 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:57.065647 31992 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:21:57.089403 31811 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.101104 31866 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:57.102206 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:57.100797 31987 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:21:57.107398 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:57.108989 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.109202 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:57.110491 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.126232 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.126932 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:21:57.128268 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.128909 25105 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:21:57.116611 31994 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:57.129428 31994 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:57.129794 31994 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:57.131496 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.132264 25105 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 26.
I20240426 13:21:57.118821 31936 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:21:57.133023 31936 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:57.133386 31936 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:57.133430 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.134320 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:57.135156 25022 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:57.117189 31866 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:57.136191 31866 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.138115 31866 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.142670 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:57.144543 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.146315 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.145287 25096 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:21:57.146888 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:57.148169 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 67879us
I20240426 13:21:57.151584 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.152629 25624 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:21:57.152575 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.154538 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:57.147756 24996 connection.cc:232] RPC call timeout handler was delayed by -1.3429502740000316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
I20240426 13:21:57.157307 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:57.158195 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.157263 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.159792 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.160555 31892 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.160430 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:57.161969 25104 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:57.148753 31872 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:57.163326 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.158403 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.102799 32037 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.083308 32041 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.159093 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.173847 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:57.176759 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:21:57.171265 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.174384 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.179432 32031 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:21:57.162289 32033 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:57.180205 32033 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.180569 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:57.180900 31983 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.000s
I20240426 13:21:57.185561 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.190518 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:57.173437 31872 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:57.196326 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:21:57.202199 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3260) took 2898 ms (client timeout 1561 ms). Trace:
I20240426 13:21:57.204771 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:57.199820 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:57.210767 25617 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:21:57.212147 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.184904 25619 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:21:57.206175 25624 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:57.208588 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.215175 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:57.094308 31984 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:57.209761 24952 rpcz_store.cc:269] 0426 13:21:54.303433 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.303613 (+   180us) service_pool.cc:225] Handling call
0426 13:21:57.202006 (+2898393us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:57.202115 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":918493952}
I20240426 13:21:57.352079 25616 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20240426 13:21:57.353078 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
W20240426 13:21:57.354182 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 139078us
I20240426 13:21:57.179939 32031 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:57.208979 31983 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:57.908139 32037 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:21:57.903970 31994 consensus_meta.cc:287] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.774s	user 0.008s	sys 0.000s
I20240426 13:21:57.494113 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:57.967020 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.968259 25100 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:57.968598 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:57.969231 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.966594 31994 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:57.957006 31979 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.906s	user 0.009s	sys 0.000s
I20240426 13:21:57.965538 32033 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.969471 25099 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:57.969753 32031 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:21:57.442023 31809 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:57.970137 25031 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:57.501166 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:21:57.970695 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.067719 25096 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: [17.9-20.10]
I20240426 13:21:57.206154 31936 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:57.540197 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.027s	sys 0.039s
W20240426 13:21:58.075953 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.411s	user 0.030s	sys 0.042s
W20240426 13:21:57.810005 31969 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.802s	user 0.008s	sys 0.000s
I20240426 13:21:58.078949 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.972121 31983 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:57.607514 25617 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:21:57.934428 32041 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:57.803337 31975 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.796s	user 0.006s	sys 0.000s
W20240426 13:21:58.105876 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.387s	user 0.038s	sys 0.073s
W20240426 13:21:57.945415 31992 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.844s	user 0.009s	sys 0.000s
I20240426 13:21:58.180948 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:57.919871 31998 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.824s	user 0.006s	sys 0.003s
W20240426 13:21:57.939005 31987 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.826s	user 0.006s	sys 0.001s
W20240426 13:21:57.965523 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 366665us
I20240426 13:21:57.530697 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:57.601912 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:57.995292 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 601323us
I20240426 13:21:58.207393 25102 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:57.995530 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1445) took 3485 ms. Trace:
I20240426 13:21:57.440399 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:58.208760 25105 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
W20240426 13:21:58.208952 25589 connection.cc:232] RPC call timeout handler was delayed by -1.3986651230000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.227s for a response
I20240426 13:21:57.967466 25022 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 26.
I20240426 13:21:57.444103 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.209702 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 970) took 1027 ms. Trace:
I20240426 13:21:58.210093 25022 rpcz_store.cc:276] 0426 13:21:57.182777 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.190019 (+  7242us) service_pool.cc:225] Handling call
0426 13:21:58.209654 (+1019635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:58.207412 25026 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 22: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:58.214622 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:58.214633 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:58.222524 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:58.207811 25618 rpcz_store.cc:276] 0426 13:21:54.510109 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.510272 (+   163us) service_pool.cc:225] Handling call
0426 13:21:54.510656 (+   384us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:54.747249 (+236593us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:57.995218 (+3247969us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54006a0ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:57.995456 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":519005568}
I20240426 13:21:58.183351 31983 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:58.065200 31979 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.213500 31998 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:57.802947 31872 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:57.971132 31984 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.070628 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:57.604931 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.250901 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
W20240426 13:21:58.250986 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 180445us
I20240426 13:21:58.251680 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 986) took 1061 ms. Trace:
I20240426 13:21:58.251953 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:57.792215 31896 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 3.772s	user 0.010s	sys 0.011s
I20240426 13:21:58.087873 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.255903 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
W20240426 13:21:58.061111 25025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.969s	user 0.005s	sys 0.011s
I20240426 13:21:58.068634 31809 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:57.646823 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.019s	sys 0.067s
W20240426 13:21:58.259267 25025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.005s	sys 0.011s
W20240426 13:21:58.259420 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.552s	user 0.019s	sys 0.079s
W20240426 13:21:58.259734 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.57019e+06us
I20240426 13:21:58.260414 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:58.106635 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.389s	user 0.039s	sys 0.073s
I20240426 13:21:58.185009 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:58.186164 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:58.263870 25098 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:21:58.180066 31514 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:21:58.197870 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.368s	user 0.008s	sys 0.020s
W20240426 13:21:58.265069 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.531s	user 0.008s	sys 0.020s
I20240426 13:21:58.210937 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.017596 32038 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.200s
I20240426 13:21:58.230165 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:58.271956 25617 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:58.279374 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:58.231099 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:58.228176 31964 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:21:58.235677 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:58.237136 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.302765 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.238595 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3399) took 1239 ms. Trace:
I20240426 13:21:58.230455 32013 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:58.245157 31994 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.102828 31969 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.246955 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.317667 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:57.781785 32030 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.030s	sys 0.054s
W20240426 13:21:57.792827 32024 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.622s	user 0.000s	sys 0.000s
I20240426 13:21:58.328330 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:58.082175 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:58.087651 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.283172 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:58.252213 25620 rpcz_store.cc:276] 0426 13:21:57.190048 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.604472 (+414424us) service_pool.cc:225] Handling call
0426 13:21:58.251615 (+647143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:21:58.183744 31975 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:58.089452 25101 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.000s	sys 0.016s
W20240426 13:21:58.300961 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47629.3us
W20240426 13:21:58.337014 25101 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.164s	user 0.000s	sys 0.017s
I20240426 13:21:58.301592 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:21:58.302532 25623 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:21:58.304314 25031 rpcz_store.cc:276] 0426 13:21:56.999165 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.088571 (+ 89406us) service_pool.cc:225] Handling call
0426 13:21:57.088898 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.066710 (+977812us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.238401 (+171691us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:58.238545 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2026275200}
I20240426 13:21:58.306071 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:58.340593 24948 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:58.244362 31798 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:21:58.307317 31869 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.146s	user 0.052s	sys 0.079s
I20240426 13:21:58.234628 31983 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.301355 31299 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:58.207856 31992 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.261286 31841 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:58.260673 31653 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:58.262387 31809 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:58.333665 25100 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 15: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:58.325248 31964 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:58.230291 32041 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
W20240426 13:21:58.324647 32030 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.395s	user 0.030s	sys 0.063s
I20240426 13:21:58.308378 31998 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
W20240426 13:21:58.307878 25097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.027s	sys 0.029s
W20240426 13:21:58.369881 25097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.028s	sys 0.029s
I20240426 13:21:58.370609 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3073) took 1017 ms. Trace:
W20240426 13:21:58.308789 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.255s	user 0.013s	sys 0.021s
I20240426 13:21:58.303882 31514 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.343291 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:58.345855 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:58.353849 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.370929 25097 rpcz_store.cc:276] 0426 13:21:57.352958 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.353076 (+   118us) service_pool.cc:225] Handling call
0426 13:21:57.353462 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:57.353729 (+   267us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:57.441145 (+ 87416us) log.cc:842] Serialized 14 byte log entry
0426 13:21:58.370324 (+929179us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:58.370372 (+    48us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:58.370411 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.370468 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.370561 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":87686,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"thread_start_us":868455,"threads_started":1}
I20240426 13:21:58.373679 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3239) took 5123 ms. Trace:
I20240426 13:21:58.375012 25617 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:21:58.376451 25619 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:21:58.374182 24956 rpcz_store.cc:276] 0426 13:21:53.246244 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.246679 (+   435us) service_pool.cc:225] Handling call
0426 13:21:53.247060 (+   381us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:53.356261 (+109201us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:53.356486 (+   225us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:53.357812 (+  1326us) log.cc:842] Serialized 26 byte log entry
0426 13:21:53.537049 (+179237us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:53.537090 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:53.537164 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:57.200899 (+3663735us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:21:58.365409 (+1164510us) raft_consensus.cc:1671] finished
0426 13:21:58.365442 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.369299 (+  3857us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6706,"raft.queue_time_us":27615,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":49644416,"thread_start_us":216929,"threads_started":2,"wal-append.queue_time_us":370297}
I20240426 13:21:58.381230 25618 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:58.381505 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:58.364184 31841 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:58.343082 31702 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
I20240426 13:21:58.376806 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:58.322762 32013 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.387194 31702 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:58.387773 31702 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.388675 32013 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.342005 31798 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.389750 31702 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.384879 31841 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:58.392937 31992 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.351079 24950 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:21:58.362103 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.369820 31998 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:58.394320 25620 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:21:58.364614 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.395246 24948 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
W20240426 13:21:58.345975 31869 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.185s	user 0.052s	sys 0.079s
I20240426 13:21:58.367686 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.383157 32030 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
W20240426 13:21:58.371870 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.318s	user 0.014s	sys 0.021s
I20240426 13:21:58.380074 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.382597 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.382416 31994 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:58.358819 32031 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.361788 31299 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.384734 31292 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:58.385890 31902 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.408424 31994 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.415772 31994 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:21:58.361290 24996 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.868s	user 0.014s	sys 0.018s
I20240426 13:21:58.212852 31987 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:21:58.325115 32033 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20240426 13:21:58.398918 24949 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:21:58.399786 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:58.401351 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 972) took 3115 ms. Trace:
I20240426 13:21:58.366406 31989 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:58.402781 24956 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:21:58.438688 25615 rpcz_store.cc:276] 0426 13:21:55.285654 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.286527 (+   873us) service_pool.cc:225] Handling call
0426 13:21:55.299853 (+ 13326us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:56.260813 (+960960us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:56.260854 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:56.261399 (+   545us) log.cc:842] Serialized 52 byte log entry
0426 13:21:56.379484 (+118085us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:21:56.379545 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:56.380297 (+   752us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:58.401072 (+2020775us) raft_consensus.cc:1671] finished
0426 13:21:58.401100 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.401288 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":581062,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":20544,"thread_start_us":1067873,"threads_started":2,"wal-append.queue_time_us":822072}
I20240426 13:21:58.413534 32031 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.377367 31653 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:58.442562 31653 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.393980 31936 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.415963 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.422561 31983 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:21:58.404542 31862 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 4.213s	user 0.053s	sys 0.140s
I20240426 13:21:58.419262 31946 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:21:58.444506 31653 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.444756 31946 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:58.407764 31292 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:58.445158 31946 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:58.445359 31292 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:58.386765 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:58.446573 31862 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 4.256s	user 0.053s	sys 0.140s
I20240426 13:21:58.437139 31964 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.411125 31299 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.451532 32010 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:21:58.451891 32010 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:58.451942 31989 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.452806 32010 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.453815 32010 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:21:58.454000 31989 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:58.421612 31969 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.422539 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2612) took 1443 ms. Trace:
I20240426 13:21:58.456868 25024 rpcz_store.cc:276] 0426 13:21:56.978735 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:56.980594 (+  1859us) service_pool.cc:225] Handling call
0426 13:21:56.991010 (+ 10416us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:57.054455 (+ 63445us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:57.054488 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:57.054527 (+    39us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:57.054557 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.422274 (+1367717us) spinlock_profiling.cc:243] Waited 63.2 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:58.422337 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.422475 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145473280}
I20240426 13:21:58.458817 31987 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:58.420825 31975 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.461508 24954 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:58.398849 31998 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:21:58.432974 24996 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.014s	sys 0.018s
I20240426 13:21:58.438761 24950 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:21:58.441733 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:58.423367 31798 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.464262 24956 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:21:58.463986 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.27717e+06us
I20240426 13:21:58.410069 31979 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.420644 31841 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.404752 31514 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.440403 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:58.424191 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:58.406245 32030 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:58.481349 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:58.484161 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:58.486917 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:58.422883 31809 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.495726 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:58.487805 32054 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20240426 13:21:58.498823 32054 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:58.499168 32054 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:58.499943 32054 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.487649 31902 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:58.489420 31883 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:21:58.506177 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:21:58.509284 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3261) took 4115 ms. Trace:
I20240426 13:21:58.507956 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:58.501178 32054 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:58.509699 24947 rpcz_store.cc:276] 0426 13:21:54.394343 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.394575 (+   232us) service_pool.cc:225] Handling call
0426 13:21:54.394880 (+   305us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:54.817751 (+422871us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:54.817797 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:54.830950 (+ 13153us) log.cc:842] Serialized 26 byte log entry
0426 13:21:54.867579 (+ 36629us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:54.867632 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.505871 (+3638239us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:58.505930 (+    59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:58.505984 (+    54us) raft_consensus.cc:1671] finished
0426 13:21:58.506000 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.509232 (+  3232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16959,"raft.queue_time_us":2394,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":2897,"spinlock_wait_cycles":971514240,"thread_start_us":19217,"threads_started":1,"wal-append.queue_time_us":167839,"wal-append.run_cpu_time_us":1617,"wal-append.run_wall_time_us":1037253}
I20240426 13:21:58.513111 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.513891 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:21:58.516688 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.519995 24956 raft_consensus.cc:2354] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:21:58.520578 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.521268 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:58.521538 24947 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:58.509709 24950 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:21:58.526547 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.527045 31292 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.527627 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.530543 31744 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:58.532290 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3394) took 1534 ms. Trace:
I20240426 13:21:58.528190 31766 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:21:58.534006 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.534806 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.536394 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.539875 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.540534 24950 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:58.540647 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:58.533330 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:21:58.536964 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:58.544150 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3069) took 1192 ms. Trace:
I20240426 13:21:58.544617 25096 rpcz_store.cc:276] 0426 13:21:57.351869 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.980380 (+628511us) service_pool.cc:225] Handling call
0426 13:21:58.067421 (+ 87041us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:58.068327 (+   906us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.543959 (+475632us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b540009c9d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:21:58.544102 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":184116992}
I20240426 13:21:58.550122 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.535617 25025 rpcz_store.cc:276] 0426 13:21:56.997722 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.077136 (+ 79414us) service_pool.cc:225] Handling call
0426 13:21:57.090536 (+ 13400us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:57.090852 (+   316us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:57.090881 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:57.091282 (+   401us) log.cc:842] Serialized 26 byte log entry
0426 13:21:58.259685 (+1168403us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:58.259722 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.259785 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:58.532090 (+272305us) raft_consensus.cc:1671] finished
0426 13:21:58.532108 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.532235 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":439,"raft.queue_time_us":594,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":1009152,"thread_start_us":1168006,"threads_started":1,"wal-append.queue_time_us":1386915}
I20240426 13:21:58.551980 31766 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:21:58.552197 31809 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.552600 31766 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:58.553128 31766 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:58.553633 31841 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.557983 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.558995 24948 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
I20240426 13:21:58.538522 24956 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
W20240426 13:21:58.562271 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.525s	user 0.019s	sys 0.030s
W20240426 13:21:58.562647 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.525s	user 0.019s	sys 0.030s
I20240426 13:21:58.563395 24948 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:21:58.563475 24950 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20240426 13:21:58.568650 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.561s	user 0.013s	sys 0.035s
W20240426 13:21:58.568969 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.561s	user 0.013s	sys 0.035s
I20240426 13:21:58.574726 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:58.575584 24954 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:21:58.580861 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:58.569348 25070 connection.cc:232] RPC call timeout handler was delayed by -1.5067300940000905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
I20240426 13:21:58.611670 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 1119) took 1493 ms. Trace:
I20240426 13:21:58.535912 31946 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:58.622745 31946 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.623627 25623 rpcz_store.cc:276] 0426 13:21:57.118653 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.118785 (+   132us) service_pool.cc:225] Handling call
0426 13:21:57.119079 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.339355 (+1220276us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.611462 (+272107us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:58.611616 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2675998464}
I20240426 13:21:58.575064 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3389) took 1595 ms. Trace:
I20240426 13:21:58.618695 31641 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:58.625093 31641 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.627756 31641 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.597997 31766 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:58.630506 25023 rpcz_store.cc:276] 0426 13:21:56.979532 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.036316 (+ 56784us) service_pool.cc:225] Handling call
0426 13:21:57.036560 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:57.036728 (+   168us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:57.037001 (+   273us) log.cc:842] Serialized 14 byte log entry
0426 13:21:58.574341 (+1537340us) log.cc:842] Serialized 14 byte log entry
0426 13:21:58.574676 (+   335us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:58.574730 (+    54us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:58.574788 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.574872 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.575001 (+   129us) inbound_call.cc:173] Queueing success response
0426 13:21:58.626270 (+ 51269us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b1800407440. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"raft.queue_time_us":11582,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":14245,"spinlock_wait_cycles":18436864,"thread_start_us":1525839,"threads_started":1}
I20240426 13:21:58.644481 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:58.645352 25096 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:58.645787 25096 raft_consensus.cc:603] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:58.647230 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:58.577723 32033 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:21:58.661091 25103 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.569s	user 0.020s	sys 0.028s
W20240426 13:21:58.661569 25103 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.570s	user 0.020s	sys 0.029s
W20240426 13:21:58.682224 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101454us
W20240426 13:21:58.702622 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55512us
I20240426 13:21:58.547621 32030 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:58.823107 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 119133us
W20240426 13:21:58.830952 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 147800us
I20240426 13:21:58.831149 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2793) took 3225 ms. Trace:
I20240426 13:21:58.836831 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:58.839011 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:58.834046 24953 rpcz_store.cc:276] 0426 13:21:55.605552 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.605995 (+   443us) service_pool.cc:225] Handling call
0426 13:21:55.654872 (+ 48877us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:56.749424 (+1094552us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.830901 (+2081477us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:58.831085 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2503724288}
I20240426 13:21:58.652194 31766 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:58.712105 32031 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:21:58.866561 31888 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:58.874378 31889 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:21:58.865473 31972 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:21:58.874457 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3390) took 1894 ms. Trace:
I20240426 13:21:58.815192 32037 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:21:58.877141 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:21:58.882701 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:58.914975 25624 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:21:58.943286 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76332.9us
I20240426 13:21:58.884450 25027 rpcz_store.cc:276] 0426 13:21:56.979829 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:57.052604 (+ 72775us) service_pool.cc:225] Handling call
0426 13:21:57.052948 (+   344us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:57.053277 (+   329us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:57.053302 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:57.053689 (+   387us) log.cc:842] Serialized 26 byte log entry
0426 13:21:58.402061 (+1348372us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:58.402106 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.402181 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:58.874189 (+472008us) raft_consensus.cc:1671] finished
0426 13:21:58.874218 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:58.874385 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3364,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10162,"thread_start_us":1348049,"threads_started":1,"wal-append.queue_time_us":1818190}
I20240426 13:21:58.879076 32031 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:58.972136 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:58.879388 31292 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:58.911129 31984 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:58.993440 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110838us
I20240426 13:21:58.955327 32031 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:21:59.036598 32066 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20240426 13:21:59.038872 32065 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20240426 13:21:59.062548 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:58.992132 32037 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:21:59.065810 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 177951us
W20240426 13:21:59.061640 31998 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.000s
W20240426 13:21:59.098258 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 126240us
I20240426 13:21:59.099221 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:59.102259 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:59.070371 31888 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:21:59.017634 31984 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.126943 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1446) took 3989 ms. Trace:
I20240426 13:21:59.116626 31873 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:21:59.126621 31888 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:59.132208 31926 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:59.138340 31926 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:59.132855 25622 rpcz_store.cc:276] 0426 13:21:55.137663 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:55.137861 (+   198us) service_pool.cc:225] Handling call
0426 13:21:55.138197 (+   336us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:55.353794 (+215597us) raft_consensus.cc:1520] Early marking committed up to index 14
0426 13:21:56.200822 (+847028us) log.cc:842] Serialized 14 byte log entry
0426 13:21:56.431006 (+230184us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:56.431074 (+    68us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:21:56.431141 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.459270 (+2028129us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b180042f2e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:21:59.126654 (+667384us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54007521d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:59.126724 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.126874 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105921,"prepare.queue_time_us":1138867,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":95261,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":49208,"spinlock_wait_cycles":502582016,"thread_start_us":844117,"threads_started":2,"wal-append.queue_time_us":303120}
I20240426 13:21:59.139639 31292 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:59.159030 31292 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:59.161680 31292 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:59.123308 32010 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20240426 13:21:59.199201 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97030.7us
I20240426 13:21:59.138609 31888 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.108788 31998 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:59.281649 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81562.7us
I20240426 13:21:59.289671 25624 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
W20240426 13:21:59.310880 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 195458us
I20240426 13:21:59.182420 32010 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:21:59.327795 25096 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:21:59.324855 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:59.200336 31995 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.218s	user 0.001s	sys 0.001s
W20240426 13:21:59.301435 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.108s	user 0.015s	sys 0.028s
I20240426 13:21:59.344362 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:59.363734 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.015s	sys 0.029s
I20240426 13:21:59.345080 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.364534 25031 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:21:59.360741 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.361446 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:59.368225 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:59.369674 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:59.376183 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.365852 25026 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:21:59.377794 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.393306 25100 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:21:59.400601 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.401981 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.402570 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:59.403887 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:59.405242 31774 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:59.400635 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:59.311200 31900 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.011s	sys 0.034s
I20240426 13:21:59.405860 31774 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:21:59.406522 31900 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.213s	user 0.011s	sys 0.034s
I20240426 13:21:59.399892 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.407771 31774 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:59.408012 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.409582 25031 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:21:59.417749 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.421056 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2808) took 3387 ms. Trace:
I20240426 13:21:59.421437 24955 rpcz_store.cc:276] 0426 13:21:56.033946 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:56.238262 (+204316us) service_pool.cc:225] Handling call
0426 13:21:56.239318 (+  1056us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:56.594941 (+355623us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:58.193523 (+1598582us) log.cc:842] Serialized 14 byte log entry
0426 13:21:59.364211 (+1170688us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:59.364420 (+   209us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:59.364464 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.420865 (+ 56401us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54000a64d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:59.420906 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.421006 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1604155,"prepare.queue_time_us":1598953,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":12182,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":817096960,"thread_start_us":1170331,"threads_started":1}
I20240426 13:21:59.428138 31928 raft_consensus.cc:465] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Not starting pre-election -- already a leader
I20240426 13:21:59.430244 31984 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:59.431685 31926 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:21:59.438206 31998 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:59.438830 31880 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:21:59.440654 31880 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:21:59.441138 31880 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:21:59.280934 32037 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.280305 31766 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:21:59.362874 32054 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.000s
I20240426 13:21:59.363289 31807 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.089s
I20240426 13:21:59.452435 31807 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:21:59.452816 31807 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:21:59.453662 31807 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.362684 32010 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:21:59.376580 31873 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.104s
I20240426 13:21:59.465900 24952 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
W20240426 13:21:59.473018 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 111688us
I20240426 13:21:59.476126 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:21:59.402565 25026 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
W20240426 13:21:59.301339 32001 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.115s	user 0.012s	sys 0.031s
I20240426 13:21:59.489909 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3268) took 1202 ms. Trace:
I20240426 13:21:59.490269 24952 rpcz_store.cc:276] 0426 13:21:58.288050 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.288243 (+   193us) service_pool.cc:225] Handling call
0426 13:21:58.321367 (+ 33124us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:59.466364 (+1144997us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.489694 (+ 23330us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:21:59.489860 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2631487872}
I20240426 13:21:59.504245 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:21:59.505184 25096 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 17
I20240426 13:21:59.454980 31807 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:59.513408 25024 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:21:59.530089 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:21:59.530985 25097 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
W20240426 13:21:59.487363 32001 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.301s	user 0.012s	sys 0.032s
I20240426 13:21:59.518853 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.488951 32005 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:21:59.516388 25027 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:21:59.552790 25025 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20240426 13:21:59.521934 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:21:59.577863 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56022.1us
W20240426 13:21:59.539821 31996 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.621s	user 0.064s	sys 0.138s
W20240426 13:21:59.608181 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.420s	user 0.025s	sys 0.059s
I20240426 13:21:59.581202 31932 raft_consensus.cc:2741] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20240426 13:21:59.633051 25101 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.263s	user 0.029s	sys 0.067s
I20240426 13:21:59.633247 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:21:59.641575 25101 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.029s	sys 0.068s
I20240426 13:21:59.600603 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:59.637650 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.299s	user 0.055s	sys 0.115s
W20240426 13:21:59.638056 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.450s	user 0.028s	sys 0.062s
I20240426 13:21:59.607282 32005 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:21:59.649107 25618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s	user 0.007s	sys 0.068s
I20240426 13:21:59.639844 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:21:59.640607 25028 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.286s	user 0.035s	sys 0.063s
W20240426 13:21:59.650238 25028 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.296s	user 0.035s	sys 0.063s
I20240426 13:21:59.650301 25099 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
W20240426 13:21:59.642719 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.304s	user 0.055s	sys 0.115s
W20240426 13:21:59.651000 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.31233e+06us
I20240426 13:21:59.642813 25100 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:21:59.651316 25618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.235s	user 0.007s	sys 0.068s
I20240426 13:21:59.652029 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:59.643530 25069 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.250s	user 0.017s	sys 0.081s
I20240426 13:21:59.643654 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3257) took 5462 ms. Trace:
I20240426 13:21:59.601708 31902 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:21:59.653362 25069 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.260s	user 0.017s	sys 0.081s
W20240426 13:21:59.653668 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.26086e+06us
I20240426 13:21:59.665170 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:59.667137 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:59.653591 24951 rpcz_store.cc:276] 0426 13:21:54.181270 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.181446 (+   176us) service_pool.cc:225] Handling call
0426 13:21:54.181993 (+   547us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:54.182175 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:54.472539 (+290364us) log.cc:842] Serialized 14 byte log entry
0426 13:21:58.188167 (+3715628us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b180008c3e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018 00007fda90330f98
0426 13:21:58.188362 (+   195us) log.cc:842] Serialized 14 byte log entry
0426 13:21:59.643100 (+1454738us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:59.643153 (+    53us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:21:59.643202 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.643439 (+   237us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.643605 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":159073,"prepare.queue_time_us":3720886,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":6004,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10351,"spinlock_wait_cycles":5992832,"thread_start_us":1735995,"threads_started":2}
I20240426 13:21:59.671005 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2551) took 4835 ms. Trace:
I20240426 13:21:59.678324 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.678515 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.682067 25624 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20240426 13:21:59.679296 25030 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:59.636999 31658 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:21:59.682176 25069 connection.cc:232] RPC call timeout handler was delayed by -1.222784448000084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
W20240426 13:21:59.605942 31996 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.694s	user 0.065s	sys 0.138s
W20240426 13:21:59.684360 25069 connection.cc:232] RPC call timeout handler was delayed by -1.1876844480000273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
I20240426 13:21:59.643375 31751 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:59.644805 31884 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:59.704578 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:59.705822 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.474s	user 0.011s	sys 0.019s
W20240426 13:21:59.712527 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.481s	user 0.011s	sys 0.019s
W20240426 13:21:59.713491 25589 connection.cc:232] RPC call timeout handler was delayed by -1.2012259580000091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
I20240426 13:21:59.714550 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:21:59.688764 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.245s	user 0.026s	sys 0.077s
I20240426 13:21:59.671394 31799 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:21:59.706352 25104 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.457s	user 0.006s	sys 0.024s
I20240426 13:21:59.704690 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:21:59.688472 25069 connection.cc:232] RPC call timeout handler was delayed by -1.1723844480000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20240426 13:21:59.720333 25104 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.471s	user 0.006s	sys 0.024s
W20240426 13:21:59.720543 25069 connection.cc:232] RPC call timeout handler was delayed by -1.1633880599999884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20240426 13:21:59.715595 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.028s	sys 0.079s
W20240426 13:21:59.721040 25069 connection.cc:232] RPC call timeout handler was delayed by -1.1358382690000326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2569s for a response
I20240426 13:21:59.721814 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3281) took 1315 ms. Trace:
W20240426 13:21:59.706079 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.383s	user 0.019s	sys 0.011s
W20240426 13:21:59.722292 25069 connection.cc:232] RPC call timeout handler was delayed by -1.064388059999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20240426 13:21:59.722508 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.399s	user 0.019s	sys 0.011s
I20240426 13:21:59.671317 31928 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:59.725085 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:59.676568 25101 rpcz_store.cc:276] 0426 13:21:54.835269 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:54.835414 (+   145us) service_pool.cc:225] Handling call
0426 13:21:54.853059 (+ 17645us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:55.160183 (+307124us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:55.160230 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:57.161773 (+2001543us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b2c0012e098. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:21:57.172910 (+ 11137us) log.cc:842] Serialized 52 byte log entry
0426 13:21:58.337446 (+1164536us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:21:59.642086 (+1304640us) log.cc:842] Serialized 14 byte log entry
0426 13:21:59.644631 (+  2545us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.644747 (+   116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:59.644801 (+    54us) raft_consensus.cc:1671] finished
0426 13:21:59.670724 (+ 25923us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.670909 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39442,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":17191,"spinlock_wait_cycles":32632576,"thread_start_us":2728453,"threads_started":3,"wal-append.queue_time_us":1650638}
I20240426 13:21:59.728564 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:21:59.682902 31887 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:59.738762 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:59.740665 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.746487 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:21:59.747146 25101 raft_consensus.cc:595] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:21:59.747416 25023 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20240426 13:21:59.748737 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:59.749171 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.749871 24955 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:21:59.749984 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.674026 31902 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.747540 25101 raft_consensus.cc:603] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:21:59.752739 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.752774 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.756042 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.757450 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3065) took 2771 ms. Trace:
I20240426 13:21:59.757845 25103 rpcz_store.cc:276] 0426 13:21:56.985688 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:56.985888 (+   200us) service_pool.cc:225] Handling call
0426 13:21:57.005365 (+ 19477us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:57.038459 (+ 33094us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:57.038501 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:57.057056 (+ 18555us) log.cc:842] Serialized 26 byte log entry
0426 13:21:58.662085 (+1605029us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:21:58.662139 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:58.662231 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:21:59.757242 (+1095011us) raft_consensus.cc:1671] finished
0426 13:21:59.757261 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.757379 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32652,"raft.queue_time_us":32795,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1570045,"threads_started":1,"wal-append.queue_time_us":2571119}
I20240426 13:21:59.760854 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:21:59.762871 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.763099 24951 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:59.763509 24955 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:21:59.766472 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.769599 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:21:59.723198 31910 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:59.772671 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.749837 24953 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:21:59.727223 31932 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:21:59.737555 31799 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:21:59.689464 31658 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:59.727429 31928 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:59.777462 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.779595 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.780817 25622 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:21:59.781615 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:21:59.779107 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:21:59.780743 25615 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.303s	user 0.038s	sys 0.061s
I20240426 13:21:59.739682 32035 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:21:59.731436 31917 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:21:59.805620 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:59.785526 25096 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:21:59.720876 31751 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:21:59.721320 31884 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.750492 31885 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.098s
I20240426 13:21:59.742352 31887 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.770897 32005 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:21:59.774628 31776 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.274s	user 0.045s	sys 0.052s
W20240426 13:21:59.864845 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44113.8us
I20240426 13:21:59.949177 24952 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:21:59.941033 24947 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20240426 13:21:59.744364 31862 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.297s	user 0.036s	sys 0.074s
I20240426 13:21:59.846170 25023 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:21:59.813107 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.722301 24949 rpcz_store.cc:276] 0426 13:21:58.406786 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.442448 (+ 35662us) service_pool.cc:225] Handling call
0426 13:21:58.442717 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.442912 (+   195us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:58.443175 (+   263us) log.cc:842] Serialized 14 byte log entry
0426 13:21:59.721464 (+1278289us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:59.721522 (+    58us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:21:59.721568 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.721645 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.721744 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":711,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":3007,"thread_start_us":1277940,"threads_started":1}
W20240426 13:21:59.809152 24956 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.231s	user 0.035s	sys 0.063s
W20240426 13:21:59.845805 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61042us
W20240426 13:21:59.946161 25615 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.038s	sys 0.072s
I20240426 13:21:59.900071 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.871721 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.923382 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.950570 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.967555 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:59.967607 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.969177 24947 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:21:59.783730 31292 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:21:59.842082 25025 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:21:59.948419 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174043us
I20240426 13:21:59.780372 31980 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:21:59.971310 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:21:59.783775 31964 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.333s	user 0.022s	sys 0.089s
I20240426 13:21:59.843849 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.930087 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:21:59.950016 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:59.978066 25031 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
W20240426 13:21:59.971772 24956 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.393s	user 0.036s	sys 0.064s
I20240426 13:21:59.898499 31461 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20240426 13:21:59.949263 31776 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.448s	user 0.054s	sys 0.056s
I20240426 13:21:59.984541 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:21:59.987533 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:21:59.983599 32028 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20240426 13:21:59.968139 31862 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.039s	sys 0.078s
W20240426 13:21:59.985708 31964 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.535s	user 0.024s	sys 0.096s
I20240426 13:21:59.894146 31873 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:21:59.990571 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:21:59.953119 32006 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:21:59.806118 32033 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:21:59.813927 31928 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:21:59.988886 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3297) took 1542 ms. Trace:
I20240426 13:21:59.999284 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:21:59.889394 32067 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.246s	user 0.002s	sys 0.000s
I20240426 13:22:00.011225 25029 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 9. Current term is 10. Ops: []
I20240426 13:21:59.985222 31980 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:00.005308 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:21:59.995790 32031 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.932s	user 0.012s	sys 0.000s
I20240426 13:21:59.970230 31292 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:21:59.990824 31461 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:00.019475 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:00.008847 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.985481 31885 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:21:59.998591 32028 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:00.009426 32006 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:00.007078 24956 rpcz_store.cc:276] 0426 13:21:58.446076 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.561153 (+115077us) service_pool.cc:225] Handling call
0426 13:21:58.561463 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.561685 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 9
0426 13:21:58.578221 (+ 16536us) log.cc:842] Serialized 14 byte log entry
0426 13:21:59.988446 (+1410225us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:21:59.988509 (+    63us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:21:59.988564 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:21:59.988678 (+   114us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:21:59.988826 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7895,"prepare.queue_time_us":16284,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":68,"raft.queue_time_us":8078,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":8888,"thread_start_us":1401793,"threads_started":1}
I20240426 13:21:59.897233 31910 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:21:59.989576 31982 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:00.014386 24948 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:22:00.002353 31880 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.561s	user 0.008s	sys 0.003s
I20240426 13:22:00.009781 31818 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:21:59.977229 31917 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:00.018816 24955 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:22:00.000233 31702 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:00.047271 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.016273 31902 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:21:59.997099 32078 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:00.027061 31998 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
W20240426 13:22:00.036774 32010 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.004s
I20240426 13:22:00.016005 32076 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.011s
I20240426 13:22:00.033572 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:00.043721 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:21:59.806006 31932 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.013614 31873 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.027143 32028 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:00.026259 31461 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:00.019136 32031 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.027842 31910 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:00.067510 24955 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20240426 13:22:00.034237 31721 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.162s	user 0.000s	sys 0.020s
I20240426 13:22:00.027426 32006 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:00.033596 31885 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:21:59.843184 32060 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.220s
I20240426 13:22:00.018883 31658 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.032809 24953 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
W20240426 13:22:00.054255 25588 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.097s	user 0.012s	sys 0.010s
I20240426 13:22:00.035640 31818 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:00.017995 31980 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:00.056896 31998 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:00.055171 31702 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:00.070927 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.080972 24955 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:21:59.905511 31641 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.096s
I20240426 13:22:00.037703 31880 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.075898 25588 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.119s	user 0.012s	sys 0.010s
I20240426 13:22:00.055912 31917 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.083864 24952 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:00.062625 32010 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.076815 31818 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:00.087641 32077 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20240426 13:22:00.084345 31998 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:00.086012 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.12898e+06us
I20240426 13:22:00.118232 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:00.122603 24955 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:00.061406 31292 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.069847 31721 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.197s	user 0.000s	sys 0.022s
I20240426 13:22:00.128602 31702 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.121922 32072 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20240426 13:22:00.134477 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.139788 24956 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:00.153430 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:00.155417 24956 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:22:00.265353 32084 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:22:00.144629 31887 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:00.264281 32082 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.187s	user 0.002s	sys 0.001s
W20240426 13:22:00.229295 32083 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20240426 13:22:00.221105 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2626) took 2101 ms. Trace:
I20240426 13:22:00.293936 25029 rpcz_store.cc:276] 0426 13:21:58.119969 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.334683 (+214714us) service_pool.cc:225] Handling call
0426 13:21:58.335011 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:00.013610 (+1678599us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.220844 (+207234us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54003af2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:00.221047 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3854536960}
W20240426 13:22:00.273838 32037 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.014s	sys 0.017s
I20240426 13:22:00.300740 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:00.280717 31859 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:22:00.281286 32060 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:00.316253 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:00.308975 32030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.010s	sys 0.021s
W20240426 13:22:00.305505 31888 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.870s	user 0.011s	sys 0.038s
I20240426 13:22:00.322561 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:00.327502 25031 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:00.328950 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:00.324189 24956 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
W20240426 13:22:00.310351 32037 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.014s	sys 0.017s
W20240426 13:22:00.333143 32030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.011s	sys 0.021s
I20240426 13:22:00.312433 31859 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:00.317054 32060 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:00.333057 31888 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.012s	sys 0.039s
I20240426 13:22:00.347108 24955 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:00.318881 31873 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:00.319932 32031 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:00.342798 31972 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.000s	sys 0.034s
I20240426 13:22:00.336776 32026 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:00.356575 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.336526 31932 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:00.319223 31658 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20240426 13:22:00.338864 31880 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.347934 31885 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:00.344316 32037 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:00.355533 31917 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.357525 32026 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:00.371001 31658 consensus_queue.cc:1225] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:22:00.360272 32010 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:00.392838 31818 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.395251 31818 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.371212 31884 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:00.376657 32026 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:00.377246 32037 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:00.354293 31972 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.000s	sys 0.034s
W20240426 13:22:00.399222 32001 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s	user 0.020s	sys 0.018s
W20240426 13:22:00.420015 32087 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:22:00.370745 32060 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.422072 32006 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.422981 32060 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:22:00.418205 31807 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.877s	user 0.015s	sys 0.007s
I20240426 13:22:00.363235 31859 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.423935 32028 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.430366 32088 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20240426 13:22:00.417625 31928 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.424165 31980 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.450860 32028 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.451097 31928 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.451638 31980 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.429338 31910 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.429569 31751 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.432655 32037 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:00.418874 32001 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.020s	sys 0.019s
I20240426 13:22:00.425189 31461 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.463002 32076 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:22:00.461086 31859 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:00.467978 32078 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:00.473305 32076 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:00.465890 32037 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:00.500560 32076 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.501781 32076 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:22:00.503172 32078 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.533951 24951 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.007s	sys 0.007s
W20240426 13:22:00.534794 24951 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.007s	sys 0.007s
I20240426 13:22:00.541805 24950 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:00.543867 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.543402 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.544806 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:00.545431 24953 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:22:00.361231 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 340546us
I20240426 13:22:00.545997 24949 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:00.545365 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.547019 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.547839 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:00.548336 25029 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
W20240426 13:22:00.549923 25024 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.003s	sys 0.020s
W20240426 13:22:00.550110 25022 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.009s	user 0.017s	sys 0.023s
W20240426 13:22:00.550348 25024 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.003s	sys 0.020s
W20240426 13:22:00.550442 25022 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s	user 0.017s	sys 0.023s
W20240426 13:22:00.551120 25026 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.012s	sys 0.012s
W20240426 13:22:00.551355 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.002s	sys 0.042s
W20240426 13:22:00.551506 25026 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.012s	sys 0.012s
W20240426 13:22:00.551734 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.003s	sys 0.042s
W20240426 13:22:00.551147 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.010s	sys 0.015s
W20240426 13:22:00.553517 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.010s	sys 0.015s
I20240426 13:22:00.543357 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.554922 24952 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:00.547304 24948 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:00.558446 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:00.559887 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.560284 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:00.560454 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:00.580487 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.000s	sys 0.027s
W20240426 13:22:00.580718 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.010s	sys 0.018s
W20240426 13:22:00.581075 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.000s	sys 0.028s
W20240426 13:22:00.581724 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.010s	sys 0.003s
W20240426 13:22:00.582006 25615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.004s	sys 0.015s
W20240426 13:22:00.582186 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.010s	sys 0.003s
W20240426 13:22:00.582473 25615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.004s	sys 0.015s
W20240426 13:22:00.584352 25104 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.011s	sys 0.022s
W20240426 13:22:00.584875 25104 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.011s	sys 0.022s
W20240426 13:22:00.585217 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.002s	sys 0.005s
I20240426 13:22:00.585310 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 995) took 2127 ms. Trace:
W20240426 13:22:00.585817 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.002s	sys 0.005s
I20240426 13:22:00.586429 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2579) took 2397 ms. Trace:
I20240426 13:22:00.586011 25615 rpcz_store.cc:276] 0426 13:21:58.457399 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.460141 (+  2742us) service_pool.cc:225] Handling call
0426 13:21:58.460576 (+   435us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.460819 (+   243us) raft_consensus.cc:1520] Early marking committed up to index 17
0426 13:21:59.968554 (+1507735us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.584151 (+615597us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.584508 (+   357us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.584586 (+    78us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:00.584800 (+   214us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.584952 (+   152us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.585216 (+   264us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1830992,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":65,"thread_start_us":2104651,"threads_started":2}
W20240426 13:22:00.586349 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 582702us
I20240426 13:22:00.588243 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.589516 25096 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:00.595057 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:00.589244 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:00.596971 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:00.599345 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:00.580473 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.004s	sys 0.012s
W20240426 13:22:00.601498 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.005s	sys 0.012s
I20240426 13:22:00.587033 25104 rpcz_store.cc:276] 0426 13:21:58.189395 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.201180 (+ 11785us) service_pool.cc:225] Handling call
0426 13:21:58.201561 (+   381us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:21:58.201936 (+   375us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:58.201974 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:21:58.202522 (+   548us) log.cc:842] Serialized 26 byte log entry
0426 13:21:59.720909 (+1518387us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:00.585366 (+864457us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.585921 (+   555us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.586048 (+   127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:00.586133 (+    85us) raft_consensus.cc:1671] finished
0426 13:22:00.586144 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.586327 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":569488,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":4690,"raft.queue_time_us":1452,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":10623,"thread_start_us":2313841,"threads_started":2,"wal-append.queue_time_us":1750222}
I20240426 13:22:00.605702 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:00.606112 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 877519us
W20240426 13:22:00.581310 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.010s	sys 0.018s
I20240426 13:22:00.607362 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.651934 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.650996 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:00.649305 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 918319us
I20240426 13:22:00.661365 25101 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:22:00.664387 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:00.664819 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 117890us
I20240426 13:22:00.668144 31944 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:00.668901 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3414) took 1330 ms. Trace:
I20240426 13:22:00.669461 25022 rpcz_store.cc:276] 0426 13:21:59.338117 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:59.351206 (+ 13089us) service_pool.cc:225] Handling call
0426 13:21:59.482180 (+130974us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:59.482394 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:59.540426 (+ 58032us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.668324 (+1127898us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.668419 (+    95us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:00.668520 (+   101us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.668618 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.668788 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117149,"prepare.queue_time_us":58231,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"thread_start_us":1010108,"threads_started":1,"wal-append.queue_time_us":930926}
I20240426 13:22:00.672236 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:00.673319 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:00.673513 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:00.678124 31818 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:00.487818 31998 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.678684 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:00.679020 31818 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.680660 31292 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:00.681707 31998 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.672457 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3427) took 1036 ms. Trace:
I20240426 13:22:00.682497 25026 rpcz_store.cc:276] 0426 13:21:59.635594 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:59.635728 (+   134us) service_pool.cc:225] Handling call
0426 13:21:59.664951 (+ 29223us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:59.665229 (+   278us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:22:00.556452 (+891223us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.671885 (+115433us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.671979 (+    94us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:00.672078 (+    99us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.672190 (+   112us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.672385 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59795,"prepare.queue_time_us":176774,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":13129,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":905761,"threads_started":2}
I20240426 13:22:00.688777 32075 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:00.689255 32075 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:00.689631 32075 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:00.691526 31575 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:00.692297 31575 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:00.670563 31910 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.667587 31862 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:00.736687 31862 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:00.739730 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:00.740731 25029 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:00.671651 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:00.742693 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:00.664410 25618 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.976s	user 0.014s	sys 0.018s
W20240426 13:22:00.744251 25618 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.059s	user 0.014s	sys 0.018s
I20240426 13:22:00.678800 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3425) took 1049 ms. Trace:
I20240426 13:22:00.728593 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2627) took 2608 ms. Trace:
I20240426 13:22:00.490459 32006 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.730000 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3426) took 1094 ms. Trace:
I20240426 13:22:00.730502 31930 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20240426 13:22:00.660959 31702 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.016s	sys 0.019s
I20240426 13:22:00.735324 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:00.664422 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.857583 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
W20240426 13:22:00.820179 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 204122us
I20240426 13:22:00.693851 31575 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:00.818897 25025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.006s	sys 0.006s
I20240426 13:22:00.745440 24955 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:00.860993 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:00.870373 31702 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.016s	sys 0.020s
I20240426 13:22:00.873064 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.870605 25099 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:22:00.874197 25103 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
W20240426 13:22:00.866585 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 199153us
W20240426 13:22:00.882822 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 140377us
W20240426 13:22:00.879165 25025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.008s	sys 0.007s
I20240426 13:22:00.858875 25027 rpcz_store.cc:276] 0426 13:21:59.629183 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:59.634987 (+  5804us) service_pool.cc:225] Handling call
0426 13:21:59.649454 (+ 14467us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:59.649677 (+   223us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:21:59.680391 (+ 30714us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.678145 (+997754us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.678268 (+   123us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:00.678402 (+   134us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.678515 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.678738 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125840,"prepare.queue_time_us":100989,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3653,"raft.queue_time_us":125287,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"thread_start_us":867116,"threads_started":1,"wal-append.queue_time_us":817493}
I20240426 13:22:00.890357 25030 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:00.895452 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:00.897991 31818 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.898999 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:00.900002 25622 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:00.859483 25024 rpcz_store.cc:276] 0426 13:21:59.635301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:59.635414 (+   113us) service_pool.cc:225] Handling call
0426 13:21:59.652425 (+ 17011us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:59.652668 (+   243us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:21:59.721018 (+ 68350us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.729312 (+1008294us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.729441 (+   129us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:00.729550 (+   109us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.729672 (+   122us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.729931 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":204053,"prepare.queue_time_us":187646,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":2848,"raft.queue_time_us":179832,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":120,"thread_start_us":829466,"threads_started":1,"wal-append.queue_time_us":766199}
I20240426 13:22:00.859304 25028 rpcz_store.cc:276] 0426 13:21:58.120271 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.335197 (+214926us) service_pool.cc:225] Handling call
0426 13:21:58.335488 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:21:58.335706 (+   218us) raft_consensus.cc:1520] Early marking committed up to index 10
0426 13:21:59.650635 (+1314929us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.727970 (+1077335us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:00.728087 (+   117us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:00.728195 (+   108us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.728313 (+   118us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.728523 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143373,"prepare.queue_time_us":748789,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":178218,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":2197762,"threads_started":2,"wal-append.queue_time_us":833204}
I20240426 13:22:00.907135 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.909701 25022 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:00.912884 32104 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20240426 13:22:00.916996 31461 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:00.930423 31884 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:00.930960 31884 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:00.931412 31884 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:00.932790 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.930953 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.933763 25104 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:00.933866 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.935662 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.936537 25102 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:00.941051 32103 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:00.941546 32103 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:00.942886 32103 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.933897 25097 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20240426 13:22:00.944696 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:00.949857 25100 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:00.950333 31930 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:00.950788 31930 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:00.950582 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.971812 25096 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:22:00.963358 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:00.975905 25101 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:22:00.983839 32078 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20240426 13:22:00.984206 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.986263 31884 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:00.987614 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 1127) took 2608 ms. Trace:
I20240426 13:22:00.962288 32060 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:00.988124 31979 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:00.988483 31892 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:00.989365 25102 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:00.989619 31980 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:22:00.990082 31980 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:00.965632 31862 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:00.997560 32109 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:00.999308 32078 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:00.999812 32078 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.000347 31641 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:01.000578 31980 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.001150 31641 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:01.002799 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 267686us
I20240426 13:22:01.004298 31949 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:01.005183 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.005483 31949 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:01.006036 31949 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.005510 31721 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:01.001029 32078 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.015522 31461 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240426 13:22:01.021842 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:01.010171 32103 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:01.023841 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:00.988087 25618 rpcz_store.cc:276] 0426 13:21:58.378737 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:58.383433 (+  4696us) service_pool.cc:225] Handling call
0426 13:21:58.383737 (+   304us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:21:58.384079 (+   342us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:21:58.384117 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:21:58.399380 (+ 15263us) log.cc:842] Serialized 52 byte log entry
0426 13:21:59.672909 (+1273529us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:00.247228 (+574319us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:00.908068 (+660840us) log.cc:842] Serialized 14 byte log entry
0426 13:22:00.987060 (+ 78992us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:00.987181 (+   121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:00.987269 (+    88us) raft_consensus.cc:1671] finished
0426 13:22:00.987374 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:00.987546 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43697,"prepare.queue_time_us":658148,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":112127,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":236,"spinlock_wait_cycles":3385088,"thread_start_us":2416435,"threads_started":2,"wal-append.queue_time_us":1649546}
W20240426 13:22:01.028667 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 126412us
W20240426 13:22:01.033655 32116 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20240426 13:22:01.033931 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:01.038784 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54653.8us
I20240426 13:22:01.055258 25100 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
W20240426 13:22:01.045758 32115 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.085s	user 0.004s	sys 0.000s
I20240426 13:22:01.065722 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.043109 31884 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:01.065825 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.067881 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.068624 25100 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.061612 32078 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:01.043818 31949 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.079416 32006 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.082408 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.083318 25103 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 14: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:01.083870 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.084939 31702 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:01.085369 31702 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.085757 31702 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.087692 31980 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:22:01.091320 31799 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20240426 13:22:01.093847 31799 consensus_queue.cc:1225] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 19.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20240426 13:22:01.094444 31928 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:01.094805 31928 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.095959 31928 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.097522 31928 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.099625 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.103981 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 19, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 19 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.105830 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:01.106581 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 20 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:01.108628 25100 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:01.108543 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 18, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 18 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.112659 31702 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.113211 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.114221 32075 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:01.108541 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53624us
I20240426 13:22:01.116591 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:01.116948 32076 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.615s	user 0.017s	sys 0.005s
I20240426 13:22:01.117825 32107 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:01.118400 32107 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.118482 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.120055 31972 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:01.120653 31972 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.120760 31949 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.121059 31972 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.114784 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 21, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.122395 24852 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 10, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 10 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:01.120218 25069 leader_election.cc:398] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:22:01.124178 25069 leader_election.cc:401] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:01.124222 32107 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.124992 31461 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17.
I20240426 13:22:01.126540 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.127480 31461 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:01.128077 31461 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.129191 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.129118 31461 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.130281 31721 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:01.131603 24852 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 14, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 14 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.127445 25096 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:22:01.133180 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.134667 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 12 to 14, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 14 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.130822 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.136585 32075 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.136914 32060 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:01.137609 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.138890 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:01.133450 31972 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:01.120132 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65285.2us
I20240426 13:22:01.077961 31641 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.141836 32060 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:01.142320 32060 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.142748 32060 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.142796 31702 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.145558 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.147192 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.147992 31980 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:22:01.150328 32120 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20240426 13:22:01.157567 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:01.158969 32037 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.004s
I20240426 13:22:01.160202 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:01.161028 25029 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:22:01.164543 25101 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:01.164984 25100 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 12. Current term is 13. Ops: []
I20240426 13:22:01.168665 25025 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:01.156482 31641 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.172775 31641 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.175307 31641 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.175779 31292 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.176541 31292 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.151800 32075 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:01.178256 31292 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.179028 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.182462 25025 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 26. Current term is 27. Ops: []
I20240426 13:22:01.185865 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.187139 25025 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:01.189602 25623 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:01.190243 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.191049 25624 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 27: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:01.190943 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:01.191707 25622 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:01.196372 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:01.197237 25102 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:01.198046 25029 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20240426 13:22:01.198058 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:01.204440 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57370.8us
I20240426 13:22:01.208786 25620 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:01.209892 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.210417 31972 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:01.210526 25029 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.208899 31751 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:01.213531 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.214674 31751 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:01.215677 31928 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:01.215225 31751 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:01.218456 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80942us
I20240426 13:22:01.217991 31751 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.216679 31998 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:01.219321 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:01.219053 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7408112bace482693c999f678aa9d73"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:01.219872 31702 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:01.220538 25097 raft_consensus.cc:595] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:01.220825 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.221096 25097 raft_consensus.cc:603] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:01.220506 31702 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.222512 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:22:01.222715 31702 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.224267 31702 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.225965 31461 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:01.226419 31461 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:01.226773 31461 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.231721 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.232218 31972 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:01.232647 31972 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.233062 31972 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.235406 25104 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:01.238384 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.242390 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:01.243556 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78878.9us
I20240426 13:22:01.175865 31641 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.244225 25096 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
W20240426 13:22:01.244798 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 68874.1us
I20240426 13:22:01.247604 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.247951 25024 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 9
I20240426 13:22:01.247502 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:01.248358 25024 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20240426 13:22:01.249852 25024 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:22:01.249784 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 150452us
I20240426 13:22:01.249195 25029 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:01.247711 24852 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 17, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 17 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.250234 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:01.253407 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.242591 32075 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:01.264328 31641 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.265851 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.266484 31641 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.270164 31930 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.272372 31930 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.276875 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.278681 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.280216 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.280314 31751 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.281087 31751 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:01.281106 32078 raft_consensus.cc:2741] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20240426 13:22:01.282474 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.284492 32110 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.242149 32060 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.286845 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.287595 32060 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:01.288249 32110 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:01.289403 24920 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.290150 32127 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:01.267086 31980 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:01.272645 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:01.293411 24852 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.296931 25104 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:01.298154 31980 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:01.309628 31751 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:01.302675 32037 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:01.317247 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.308662 31980 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.317585 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.320153 25102 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20240426 13:22:01.320187 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:01.312971 32026 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.769s	user 0.014s	sys 0.006s
I20240426 13:22:01.330200 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.307202 32060 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20240426 13:22:01.330953 25100 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.305295 32107 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.014s
I20240426 13:22:01.333346 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.333679 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:01.334774 24852 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 15, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 15 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.336931 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.337919 25096 raft_consensus.cc:2396] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20240426 13:22:01.337927 24852 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 11 to 15, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:01.340646 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.340776 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.342020 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.342775 25022 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 27: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:01.342677 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.343006 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.341572 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:01.343927 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.344305 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.342175 25027 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:01.348748 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:01.351940 31859 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.649s	user 0.015s	sys 0.011s
I20240426 13:22:01.349148 32026 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.353277 32037 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Stepping down as leader of term 26
I20240426 13:22:01.353701 32037 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.354364 31721 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:22:01.354760 31721 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:01.354912 32026 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.354597 32037 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.355319 31721 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:01.357632 25102 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:22:01.358100 32037 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:01.359648 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.354017 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.364612 31884 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:01.365024 31884 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.351366 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.365444 31884 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.368535 31980 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.371155 25025 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:22:01.372653 31972 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.374007 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:01.329533 32103 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:01.363302 31721 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.384961 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.381691 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.374564 32076 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.388669 24954 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:01.389307 32037 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:22:01.389736 32037 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:01.390151 32037 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:01.391194 32060 raft_consensus.cc:979] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:01.391685 32060 raft_consensus.cc:1067] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:01.392333 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.394826 31702 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:01.392935 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89604us
I20240426 13:22:01.397970 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.398269 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.401698 25025 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: []
I20240426 13:22:01.401468 32075 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:01.402823 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.403235 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.403669 25024 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.404234 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.405071 31721 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:01.395943 32103 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.406472 32103 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.407567 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.395432 31702 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.408435 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.409281 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.409835 31702 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.407100 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.383034 31884 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.400655 32060 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:22:01.392277 32076 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.419797 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.420153 24951 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.419173 25104 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: [17.9-20.10]
I20240426 13:22:01.421646 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.422159 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.425760 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.426420 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:22:01.430109 25097 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:01.429219 25024 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:01.431377 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.432662 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.432569 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.434197 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.432844 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.431165 31575 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:01.436897 25615 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:01.438047 24954 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:01.436861 31859 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:01.438861 25024 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20240426 13:22:01.425091 32060 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.439533 24955 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.446605 31859 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.441128 32076 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.441229 24948 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:01.441507 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.442137 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.447595 31859 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.443148 31884 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:01.448401 24952 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:01.449002 31859 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:01.443728 31980 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.443845 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.444424 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.444355 31641 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:22:01.442730 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.447533 32060 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.451633 24953 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.449036 24950 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:01.445988 31862 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:01.443089 32078 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20240426 13:22:01.460240 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.439721 31461 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.461061 32078 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:01.461686 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.467566 25096 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:01.467909 25101 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:01.468657 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.471261 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.472020 24953 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.474416 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.474869 32076 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:01.475051 24953 raft_consensus.cc:2362] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19. Re-sending same reply.
I20240426 13:22:01.475500 32076 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.475621 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.477895 24951 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:01.466377 32078 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.479214 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.479166 32078 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.480650 32078 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:01.481045 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.481638 24952 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:22:01.483532 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.487414 32107 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:01.487659 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.476572 32037 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.487890 32107 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.488535 24952 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.476538 32076 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.490805 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.470410 24950 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:01.492753 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.468313 31980 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:22:01.497045 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.500901 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.499390 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.501801 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.502033 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:01.502408 25099 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:01.502664 25615 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:22:01.503561 25098 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:22:01.505528 24954 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
W20240426 13:22:01.505759 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83602us
I20240426 13:22:01.506614 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:01.497038 32107 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.516000 32107 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.513084 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.512367 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:01.519531 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:01.521654 31641 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:01.522018 31641 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.522598 31461 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:01.522883 31641 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.524402 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.508275 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.489935 31972 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:01.531457 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.491804 31862 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:01.532119 25102 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.542506 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.543382 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.547415 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.547467 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.542574 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.549050 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:01.551826 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.551985 25105 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20240426 13:22:01.553015 25104 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20240426 13:22:01.554358 24953 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:01.534476 31575 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:22:01.566030 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:01.563781 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102245us
I20240426 13:22:01.563017 32075 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:01.592396 32075 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:01.593205 25102 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:01.593465 32075 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:01.595364 24848 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 275) took 54069 ms (client timeout 15000 ms). Trace:
I20240426 13:22:01.595695 25099 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:01.596900 32037 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:01.596550 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.604161 25025 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20240426 13:22:01.604442 31575 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:01.568588 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.604851 31575 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.570120 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.606671 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.607662 31979 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20240426 13:22:01.608917 31979 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.611682 31884 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:01.613536 31862 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:22:01.614902 31930 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:22:01.570281 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:01.570820 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62624.8us
I20240426 13:22:01.615938 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.617285 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:01.584594 32139 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20240426 13:22:01.585181 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60995us
I20240426 13:22:01.586258 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.617766 24952 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:01.615262 31930 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:22:01.595875 24848 rpcz_store.cc:269] 0426 13:21:07.526277 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:07.526618 (+   341us) service_pool.cc:225] Handling call
0426 13:22:01.594951 (+54068333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:21:52.822427 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:21:53.089628 (+267201us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:21:53.089703 (+    75us) write_op.cc:427] Acquired schema lock
0426 13:21:53.089736 (+    33us) tablet.cc:660] Decoding operations
0426 13:21:53.091013 (+  1277us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:21:53.091110 (+    97us) write_op.cc:633] Partition lock acquired for write op
0426 13:21:53.091141 (+    31us) tablet.cc:683] Acquiring locks for 13 operations
0426 13:21:53.458266 (+367125us) tablet.cc:699] Row locks acquired
0426 13:21:53.480271 (+ 22005us) write_op.cc:260] PREPARE: finished
0426 13:21:53.522158 (+ 41887us) write_op.cc:270] Start()
0426 13:21:53.550343 (+ 28185us) write_op.cc:275] Timestamp: P: 1714137713496740 usec, L: 0
0426 13:21:53.561131 (+ 10788us) op_driver.cc:331] REPLICATION: starting
0426 13:21:53.746486 (+185355us) log.cc:842] Serialized 5115 byte log entry
0426 13:21:55.044322 (+1297836us) op_driver.cc:444] REPLICATION: finished
0426 13:21:56.277658 (+1233336us) write_op.cc:300] APPLY: starting
0426 13:21:56.277771 (+   113us) tablet.cc:1365] starting BulkCheckPresence
0426 13:21:57.034128 (+756357us) tablet.cc:1368] finished BulkCheckPresence
0426 13:21:57.034281 (+   153us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:21:57.911064 (+876783us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:21:57.911494 (+   430us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0426 13:21:57.911568 (+    74us) write_op.cc:311] APPLY: finished
0426 13:21:57.912066 (+   498us) log.cc:842] Serialized 121 byte log entry
0426 13:21:59.729991 (+1817925us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:59.750671 (+ 20680us) write_op.cc:446] Released schema lock
0426 13:21:59.783577 (+ 32906us) write_op.cc:340] FINISH: Updating metrics
0426 13:21:59.862464 (+ 78887us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:21:59.862497 (+    33us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":55618,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":1169933,"apply.run_cpu_time_us":227986,"apply.run_wall_time_us":3585298,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58493,"num_ops":13,"prepare.queue_time_us":467113,"prepare.run_cpu_time_us":15327,"prepare.run_wall_time_us":980056,"raft.queue_time_us":180782,"raft.run_cpu_time_us":4623,"raft.run_wall_time_us":318225,"replication_time_us":1466594,"thread_start_us":2361792,"threads_started":5,"wal-append.queue_time_us":2060924,"wal-append.run_cpu_time_us":22700,"wal-append.run_wall_time_us":3054784}]]}
I20240426 13:22:01.602133 25097 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20240426 13:22:01.565804 31641 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:01.569011 32127 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:01.605273 31575 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.615120 25025 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20240426 13:22:01.622383 25100 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:01.634645 31980 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:01.646541 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:01.650318 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43716.3us
I20240426 13:22:01.641145 31979 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:22:01.664422 31930 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:01.665940 31930 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:22:01.666337 31980 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:01.666803 31980 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:01.668195 31902 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20240426 13:22:01.669229 31292 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:01.669641 31292 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.670549 31292 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.676982 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:01.679236 31461 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:22:01.647636 31751 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:01.680349 31461 raft_consensus.cc:2741] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:22:01.680521 31721 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:01.680894 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:01.681126 31461 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:01.681452 24951 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:01.681670 31461 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:01.681859 24951 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.682420 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:01.683656 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97532.9us
I20240426 13:22:01.684680 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.686422 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:01.687311 31702 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.687675 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:01.671970 31292 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.690698 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.688500 32026 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.687912 31702 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:22:01.696913 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.696734 32026 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.698927 32026 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.696569 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:01.700209 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44485us
I20240426 13:22:01.700412 24948 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:01.700933 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.702247 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.707137 31979 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:01.707005 31575 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.696736 31702 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:01.709167 31702 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:01.702447 24951 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.711659 24953 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 20. Ops: []
I20240426 13:22:01.667807 31980 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.713003 31575 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.714116 24951 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:01.718112 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62768.2us
I20240426 13:22:01.720345 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.688583 31751 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:01.721510 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.722473 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:01.677418 32127 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:01.730628 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.735028 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.725420 31980 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.738281 32026 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
W20240426 13:22:01.744140 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67115.7us
I20240426 13:22:01.746878 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:01.752624 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65037.9us
I20240426 13:22:01.748066 31884 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20240426 13:22:01.754166 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98023.6us
I20240426 13:22:01.734514 31751 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.756500 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.756621 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.731986 31702 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.747592 32026 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:01.768801 32026 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:01.769274 31702 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:01.765857 31751 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.766291 32110 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.770648 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:01.767320 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.747916 31575 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20240426 13:22:01.776938 31461 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.777885 31575 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:01.778345 31461 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:01.778426 31575 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.778903 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.741048 32127 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
I20240426 13:22:01.781406 32110 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20240426 13:22:01.784922 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82997.7us
I20240426 13:22:01.785660 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.779892 31575 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.781420 32147 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:01.787902 32147 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:01.788076 31575 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:01.782533 31751 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.788563 32147 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:01.795652 32127 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:01.796258 32127 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:01.797398 32127 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.799002 25025 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:01.808909 31930 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:01.811147 31859 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:01.813021 32127 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.826319 31902 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20240426 13:22:01.831130 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.832548 31930 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:01.835759 31884 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:01.836021 31930 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.836236 31862 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:01.843999 31862 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:22:01.844228 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97423.7us
I20240426 13:22:01.836833 32107 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20240426 13:22:01.845010 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.846244 32149 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:01.846649 32149 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.847159 32149 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:01.846581 31862 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.847965 31862 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.836091 31859 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:01.848680 31859 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.849476 31859 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.850767 31859 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:22:01.852488 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66882.4us
I20240426 13:22:01.853502 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.854463 32107 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:01.855168 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.864766 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.865613 32110 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.865983 31930 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:01.866778 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52413.6us
I20240426 13:22:01.867574 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.868305 32107 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:22:01.866012 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.869261 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.870626 31292 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.871695 25101 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:22:01.872064 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.872566 32110 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:01.873072 32110 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.872426 31292 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.873765 31292 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:01.874219 31292 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:01.873962 32110 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.875164 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.875000 31292 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.876464 25097 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:22:01.871317 31641 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20240426 13:22:01.878469 31979 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:01.879230 32152 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:01.878479 32147 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.880247 25097 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 10. Current term is 11. Ops: []
I20240426 13:22:01.880939 25025 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20240426 13:22:01.881314 25025 pending_rounds.cc:76] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 11
I20240426 13:22:01.881564 25025 pending_rounds.cc:97] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20240426 13:22:01.881767 31979 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:22:01.881829 25025 raft_consensus.cc:2866] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:01.882357 31979 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:01.883180 31979 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:01.884362 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.884480 31979 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:01.884538 32107 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:22:01.885573 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.886808 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.888082 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.890697 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:01.895601 31751 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:01.895915 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:01.889389 31930 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:01.896593 31930 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:01.896972 31930 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:01.897022 31751 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.902110 25026 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 13. Current term is 14. Ops: []
I20240426 13:22:01.902433 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:01.902539 31751 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:01.903666 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:22:01.901166 32078 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:22:01.906504 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.910588 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:01.905141 32078 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:01.911808 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:01.881422 32147 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:01.913882 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:01.901311 31884 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:22:01.927203 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.928411 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:01.917291 32103 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.920580 31884 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:01.923610 32107 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:01.929685 32152 consensus_queue.cc:1225] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20240426 13:22:01.938025 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.929124 31980 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:01.937014 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.922084 32147 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20240426 13:22:01.932762 32076 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:01.945263 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.936236 31884 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:01.946636 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:01.941352 31862 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:22:01.940971 31702 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:22:01.953380 31944 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.953621 31862 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:01.979880 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 112377us
I20240426 13:22:01.977048 31862 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:01.949113 31972 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:01.942090 32078 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:01.947743 32107 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:01.998031 25030 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:22:02.002105 32075 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Stepping down as leader of term 13
W20240426 13:22:02.000082 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49373.6us
I20240426 13:22:02.002828 31980 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:22:01.988317 31702 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:01.988003 31930 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:02.022215 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84226.6us
W20240426 13:22:02.024857 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 114361us
I20240426 13:22:01.948945 32037 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.069s
I20240426 13:22:02.032337 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.033234 31930 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.031924 31702 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20240426 13:22:02.030520 31944 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20240426 13:22:02.010119 31902 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:02.021711 31980 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:02.017256 32075 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:01.950954 32026 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.048889 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.052657 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.051570 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.053524 25103 raft_consensus.cc:2362] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17. Re-sending same reply.
I20240426 13:22:02.056746 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.057133 25025 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.058228 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.058820 25096 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:02.059635 31944 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:02.060031 31944 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.062707 31944 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.063064 32075 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.061518 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:02.064280 25031 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:22:02.064456 32075 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:02.061822 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.069007 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.048815 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.071141 25098 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.071249 25096 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:02.073493 32149 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.074076 25100 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20240426 13:22:02.074167 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.075434 32149 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:02.076665 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54121.6us
I20240426 13:22:02.077167 32165 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.077490 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.077733 32165 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.078574 32153 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:02.078822 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.079206 32153 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.040956 32107 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20240426 13:22:02.042326 32076 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.050s
I20240426 13:22:02.079792 32153 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:02.044574 31902 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.081640 31902 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.082809 31902 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:02.079838 32165 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.086885 31862 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.088613 31980 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.090582 31980 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:02.091296 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:02.093531 32026 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.093732 31859 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:02.094372 31859 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.100978 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:02.102224 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.102789 24954 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:02.069916 25105 raft_consensus.cc:2362] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10. Re-sending same reply.
W20240426 13:22:02.105068 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 198662us
I20240426 13:22:02.106256 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:02.108800 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.110489 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:02.111152 31575 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:02.130139 24954 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
W20240426 13:22:02.130718 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87852.1us
I20240426 13:22:02.134155 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.137023 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.118821 32147 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.130373 31575 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.188609 25099 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:02.192492 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.133976 32076 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.121368 31859 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.167421 31575 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:02.196355 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63976us
I20240426 13:22:02.233534 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:02.228562 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 94452.8us
I20240426 13:22:02.193347 25100 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
W20240426 13:22:02.231395 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 106514us
W20240426 13:22:02.190138 32131 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:22:02.169036 31979 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:02.236459 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.214471 31862 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.196722 32147 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.197026 31930 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20240426 13:22:02.240612 32076 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.321882 31930 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:02.322350 31930 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.322873 32076 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.194885 32110 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:02.237270 32127 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:02.323833 32127 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:02.324719 32127 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.326792 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.245749 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.327795 24953 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:22:02.328197 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82538.6us
I20240426 13:22:02.246320 32165 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.330605 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.331394 32155 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:02.331600 32147 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:02.248989 25099 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20240426 13:22:02.359056 25099 pending_rounds.cc:76] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 13
I20240426 13:22:02.359354 25099 pending_rounds.cc:97] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20240426 13:22:02.359612 25099 raft_consensus.cc:2866] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:02.250090 24954 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:22:02.217340 31944 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:22:02.256567 32167 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.134s	user 0.005s	sys 0.000s
I20240426 13:22:02.257433 25100 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:22:02.260264 31575 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.363734 31575 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.267184 31751 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.364415 31751 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:02.272401 32164 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20240426 13:22:02.273151 32169 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:02.276587 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:02.370465 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93950us
I20240426 13:22:02.285666 32162 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.287470 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 20. Ops: []
I20240426 13:22:02.294149 32176 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
W20240426 13:22:02.294250 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57864.7us
I20240426 13:22:02.373672 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.374265 25024 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:02.301939 31884 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.318581 32153 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.323382 31930 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.247596 25104 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:02.402591 25104 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.334079 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.331391 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:02.331048 32165 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.356376 25615 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:22:02.375429 32174 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:02.375382 32147 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:02.377748 32176 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.379441 24954 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:02.411195 32165 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.388533 25027 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.411608 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:02.389007 25623 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20240426 13:22:02.412164 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78173.9us
I20240426 13:22:02.411775 32176 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.413529 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.389425 24955 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:02.389936 25618 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:02.414510 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.390337 25621 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:02.390820 31751 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.391359 25616 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20240426 13:22:02.417025 31575 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:02.417557 31575 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:02.417726 25027 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:02.406643 25104 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.419878 25104 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:22:02.392132 32162 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.392504 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.368897 32103 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.392961 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.393343 25031 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:02.394279 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.431190 25105 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.392170 24953 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20240426 13:22:02.400463 25102 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: []
I20240426 13:22:02.401036 25024 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20240426 13:22:02.404664 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.433012 25026 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:02.408841 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:02.414206 31292 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:02.420742 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.429320 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.434085 25100 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20240426 13:22:02.401785 31884 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:02.434725 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.440053 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.436621 31292 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:02.440821 31292 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.441296 31292 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:02.441359 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.435617 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.437860 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.438011 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.438341 32181 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:02.439023 31944 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:02.417883 31575 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.450443 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:02.445216 25099 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:02.450906 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:02.453173 25101 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 16: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:02.429719 32174 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:02.439826 31930 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:02.449581 32060 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:22:02.440917 32127 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:02.450209 31862 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:22:02.429082 32075 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20240426 13:22:02.454818 31751 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:22:02.454568 31575 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:02.454931 31461 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.022s
I20240426 13:22:02.461424 31641 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:02.487434 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.492779 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.440062 32103 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.492184 25104 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
W20240426 13:22:02.497500 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122809us
I20240426 13:22:02.500727 31292 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.507993 24954 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:02.513505 31292 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.476462 32175 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:22:02.472427 32165 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:02.513643 24955 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:02.470693 32104 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:02.485410 32127 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.520736 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.525799 32171 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:02.526167 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.526307 32171 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:02.526870 25104 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:22:02.526991 32026 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:02.527498 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.484432 32037 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20240426 13:22:02.531924 31292 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:02.527223 32171 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.476557 31884 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20240426 13:22:02.539162 24954 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 15.
W20240426 13:22:02.521040 31972 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20240426 13:22:02.541632 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66075.1us
I20240426 13:22:02.507977 31751 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:02.543924 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.542312 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.547122 31292 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.547286 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.495381 31944 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20240426 13:22:02.472362 32166 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.068s
I20240426 13:22:02.529047 32127 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:02.551153 32185 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:02.561498 32156 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.562197 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.566766 24953 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:02.566483 32156 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:02.420356 32107 consensus_queue.cc:1159] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 20 status { last_received { term: 18 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:22:02.568192 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.570076 32103 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.571882 31721 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:02.572307 31721 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.572724 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:02.573155 31721 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.573103 32156 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.574477 31721 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:02.574407 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.575193 25099 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:02.576354 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:02.577132 32156 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:02.577580 32156 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:02.577018 31751 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.577931 32156 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:02.562906 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.579596 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.563125 24956 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 14. Current term is 15. Ops: [13.10-14.11]
W20240426 13:22:02.580612 32182 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20240426 13:22:02.565891 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.551880 31292 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.581666 32078 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:22:02.582310 32078 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:02.582720 32162 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.583462 32153 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.583114 32078 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:02.587402 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78628.6us
I20240426 13:22:02.596303 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.545976 32165 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:02.602684 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.604279 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:02.604738 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.604868 24950 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:02.608291 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.609690 25621 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:02.609766 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.610751 24953 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:02.528483 32104 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.614014 25025 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 24: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:02.614444 25621 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:02.601047 31702 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:22:02.622828 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:02.624275 25621 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:22:02.628669 25025 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:02.620332 32173 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:22:02.533155 32174 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.629586 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:02.632966 24952 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20240426 13:22:02.635301 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:02.636060 25623 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:02.624675 32110 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20240426 13:22:02.615129 32181 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:02.665444 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.680500 24951 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:02.682914 24950 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:02.684103 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.615260 32076 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
W20240426 13:22:02.692198 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56858us
W20240426 13:22:02.692682 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63701.1us
I20240426 13:22:02.615705 32026 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20240426 13:22:02.693563 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.696416 25026 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.710368 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:02.713784 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:02.652448 32181 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.664943 32153 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.665136 32110 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election won for term 24
W20240426 13:22:02.729094 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63722.5us
I20240426 13:22:02.691998 31902 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:02.752516 31292 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.754968 25623 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:22:02.710146 32147 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.721392 32075 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.685227 31862 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.778880 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:02.710935 31980 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.770838 25105 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:22:02.753000 31292 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.773847 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:02.733749 32110 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.732841 32169 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:02.794688 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.796936 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.798908 25099 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:02.798908 25097 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20240426 13:22:02.802661 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.743427 31641 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.803051 31292 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.803582 31292 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:02.805531 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:02.725908 31461 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.057s
I20240426 13:22:02.803763 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:02.803926 31292 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:02.808807 31292 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:02.805481 25103 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:02.806281 25105 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:02.817694 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:02.818519 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:02.818562 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69399.5us
I20240426 13:22:02.807296 32110 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:02.785598 32189 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:02.852195 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:02.807111 32169 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.808949 32185 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20240426 13:22:02.859714 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56016us
I20240426 13:22:02.803911 31641 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.800838 31702 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20240426 13:22:02.771165 32153 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.869196 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.794577 32103 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20240426 13:22:02.788535 32147 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.878834 24951 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:02.788738 31980 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:02.866359 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 96063.8us
W20240426 13:22:02.878758 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62764.8us
I20240426 13:22:02.869580 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.788261 32075 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.908004 32075 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.902081 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:02.865590 32169 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:02.808473 32155 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20240426 13:22:02.870458 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:02.916802 24950 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:02.910306 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:02.859488 32196 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:02.917443 31292 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.917898 25621 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:02.913895 31859 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:22:02.913651 32075 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:02.925958 31859 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:02.920116 32196 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:02.926671 32196 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:02.927556 31292 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.905071 31930 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:02.926251 32075 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.914952 32176 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:22:02.872304 32173 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.050s
W20240426 13:22:02.946465 32195 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20240426 13:22:02.949651 32203 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20240426 13:22:02.904657 32191 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.271s
I20240426 13:22:02.950758 32200 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20240426 13:22:02.915767 32104 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:02.952358 32186 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.292s	user 0.000s	sys 0.001s
I20240426 13:22:02.914151 31862 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.963366 31292 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.857311 32181 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.963987 31859 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:02.965107 32153 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:02.972402 32176 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:02.965768 32162 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20240426 13:22:02.973167 31980 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:02.976625 32075 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:02.922657 31884 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:02.978062 31292 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:02.936072 31930 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.000741 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:03.006767 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 188323us
W20240426 13:22:03.006136 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 139309us
I20240426 13:22:03.008451 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.011765 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.013268 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.017956 25026 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:03.018121 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.020491 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.022747 32155 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:03.024381 32216 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:03.024807 32216 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:03.025988 25026 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:03.026492 25026 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:03.025815 32216 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.028627 24950 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20240426 13:22:03.028734 25098 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20240426 13:22:03.027807 32216 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:02.988411 32173 consensus_queue.cc:1225] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:22:03.034463 31944 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:03.040748 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.027422 25026 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.048311 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.050413 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.053432 24950 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20240426 13:22:03.010087 31930 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:03.059096 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.059494 31292 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.048466 31859 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:03.069026 24951 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:22:03.069351 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 288925us
I20240426 13:22:03.076727 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.077356 25024 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:03.028348 31884 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.079371 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:03.060220 32147 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.091470 31461 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:03.091996 31461 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.096101 32075 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:03.096799 32075 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.081898 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:03.085664 31575 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.000s
I20240426 13:22:03.088358 32165 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:03.088897 32176 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.090066 32217 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:03.096716 31461 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.052215 32166 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20240426 13:22:03.115830 25024 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:03.116462 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:03.120922 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:03.114466 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:03.132257 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 131581us
I20240426 13:22:03.133409 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:03.135677 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69071.5us
I20240426 13:22:03.111747 32217 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.136914 32217 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:03.138859 32194 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:03.132872 31980 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:03.140295 32201 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:03.140794 32201 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:03.140457 32196 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.141204 32201 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.141568 32175 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:03.143803 32191 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:03.144222 32191 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.144369 32219 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:03.145128 32191 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.146371 32191 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.113911 32165 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:03.154194 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:03.161324 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.097477 32075 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.167127 25031 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20240426 13:22:03.168826 31721 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:03.169215 31721 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.149219 31461 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:03.116779 32147 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.049s
I20240426 13:22:03.151557 32166 consensus_queue.cc:1225] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 22.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20240426 13:22:03.171231 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:03.139722 32215 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:03.116504 31575 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.167299 32103 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.188143 32168 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:03.168071 32196 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.192333 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:03.201887 25030 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:03.165627 31702 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20240426 13:22:03.167467 31944 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20240426 13:22:03.181593 32176 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.188572 32231 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:03.197840 32168 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:03.272950 32231 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:03.273756 32224 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20240426 13:22:03.274817 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.197824 32171 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20240426 13:22:03.270833 32076 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.281658 32168 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:03.281435 32196 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:03.307296 32156 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.729s	user 0.007s	sys 0.004s
I20240426 13:22:03.291160 32078 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:03.303952 32194 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.272578 32176 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.350625 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.351433 25103 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:22:03.351570 32231 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.350253 32196 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.352838 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:03.353830 32197 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.547s	user 0.000s	sys 0.011s
I20240426 13:22:03.354173 32196 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.354693 32213 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:22:03.355166 32213 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.355408 32232 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.356422 32189 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:03.356818 32189 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:03.357563 32189 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.358181 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.362767 25026 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:03.363018 32169 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.365274 32169 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:03.365581 32234 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20240426 13:22:03.366883 32221 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20240426 13:22:03.338215 32156 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.369740 32156 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.371062 25621 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20240426 13:22:03.373049 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.373896 25024 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:03.376348 31972 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:03.376735 31972 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.377487 31972 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:03.381178 32237 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:22:03.381816 31575 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:03.383365 32229 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20240426 13:22:03.337056 32147 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.385610 32147 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.386265 32181 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:03.387452 32147 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:03.390115 32222 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.302s	user 0.002s	sys 0.000s
I20240426 13:22:03.190902 32215 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.355573 32213 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:03.394922 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:03.356000 32232 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.397389 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.400641 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:03.423074 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.403728 25024 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:03.412456 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.424170 25103 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:22:03.425824 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:03.314607 32076 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.398563 31884 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.393405 32075 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.434590 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.440402 24947 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 17: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:03.444976 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.444496 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:03.432123 32174 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.034s	sys 0.021s
I20240426 13:22:03.529281 32107 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:03.432966 32189 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:03.439189 32162 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:03.442319 32241 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.447757 24949 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:03.449460 24951 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:03.542122 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:03.534277 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 342046us
W20240426 13:22:03.533936 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64204.5us
I20240426 13:22:03.544447 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:03.552263 25099 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:03.530556 32188 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.128s
I20240426 13:22:03.432701 32104 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.346619 32194 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:03.534957 32216 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.000s
I20240426 13:22:03.562229 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.563436 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.532626 32218 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:03.536846 32174 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.036s	sys 0.025s
I20240426 13:22:03.574044 32232 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.575779 32176 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.580369 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:03.586970 25030 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 24: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:03.568898 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:22:03.532719 32147 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.584546 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:03.564791 32221 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20240426 13:22:03.558625 32162 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:03.564124 32127 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.962s	user 0.011s	sys 0.000s
I20240426 13:22:03.563805 31702 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.558557 32173 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:03.564224 32103 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:03.584962 31862 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.606796 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.582137 32219 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:03.641992 32219 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:03.605432 32162 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:03.607838 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.607456 25023 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:03.611439 32076 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.585144 32226 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.615234 32215 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.593212 32218 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:03.616262 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66981us
W20240426 13:22:03.616425 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56146.8us
I20240426 13:22:03.616758 32169 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240426 13:22:03.618788 32110 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.757s	user 0.009s	sys 0.000s
I20240426 13:22:03.600689 32147 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.629523 32194 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.571210 32104 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:03.633230 31930 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.000s
I20240426 13:22:03.568120 32241 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.636972 31979 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.637621 31702 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.616477 31862 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.627645 32168 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.640635 32173 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.649911 25030 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 27.
I20240426 13:22:03.597291 32232 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.532676 32156 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.643141 32219 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.674029 25026 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20240426 13:22:03.686566 24947 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20240426 13:22:03.623454 32217 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.667192 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:03.688028 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.689954 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.711843 25621 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 27.
I20240426 13:22:03.691167 25025 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:22:03.691614 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.712513 25616 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
W20240426 13:22:03.707003 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.013s	sys 0.017s
I20240426 13:22:03.708657 24949 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20240426 13:22:03.607404 32127 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.704773 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:03.714304 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.014s	sys 0.017s
I20240426 13:22:03.715404 24948 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:22:03.739476 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:03.739490 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75031.1us
I20240426 13:22:03.694450 31930 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.745851 31930 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.748996 32256 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:03.749426 32256 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.750162 32256 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.717957 32232 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.751943 32165 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.753068 32165 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.692960 32210 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20240426 13:22:03.756595 31721 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.757774 31721 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:03.760198 32254 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
W20240426 13:22:03.761221 32258 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20240426 13:22:03.762732 32127 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.763370 25615 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:03.691000 32110 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.707360 31979 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.767365 31979 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.768324 32176 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:03.768723 32176 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.769026 32147 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.769630 32176 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.770411 32104 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:03.742314 32171 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:03.771522 32171 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.759876 32107 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:22:03.770643 32201 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.629s	user 0.007s	sys 0.001s
I20240426 13:22:03.773561 32201 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.773591 32171 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.777033 31862 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.777213 32201 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.716280 32156 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.779284 32156 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.749351 32168 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:03.784139 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 44753.4us
W20240426 13:22:03.789309 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51522us
I20240426 13:22:03.673720 32226 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.802641 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.789696 32103 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:03.793797 32194 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20240426 13:22:03.808801 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.810787 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.811800 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.812444 25621 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:03.813637 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.791024 32026 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20240426 13:22:03.815167 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:03.818022 32255 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:22:03.818225 32209 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:03.819456 24955 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:03.820143 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:03.807006 31972 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:03.807281 32215 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:03.825850 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.827898 32218 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.829416 32173 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.830241 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.830998 25101 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:03.841573 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.811925 32194 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.842191 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.826354 25099 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:22:03.851007 25105 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:03.814742 32060 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.032s
I20240426 13:22:03.853374 32226 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.854652 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.807610 32103 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.816164 31575 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:03.847312 32176 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:03.848143 32218 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:03.814535 31862 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:03.814397 32225 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:03.789625 32213 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.855182 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.857504 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.859287 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.851743 32194 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:03.859844 32168 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:22:03.864157 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.858597 32218 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.861076 32060 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:03.864809 32168 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.867978 31862 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:03.861029 32260 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:03.875121 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:03.872972 32213 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.873315 32103 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.856923 32254 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20240426 13:22:03.873659 31702 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.862561 32156 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:03.753273 32232 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.869899 32218 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:03.880823 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:03.884189 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:03.884414 25623 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:03.884758 25101 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:03.878631 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.883597 32110 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:03.890205 25624 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:03.893612 31862 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:03.894495 31862 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.876488 32168 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:22:03.877207 32217 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.882284 32156 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:03.867039 32209 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20240426 13:22:03.909330 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:03.913301 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.915498 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.918183 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:03.904948 31902 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.924716 25618 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 18: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:03.925339 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.927628 25623 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20240426 13:22:03.910297 32216 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:03.923532 32198 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:03.907702 32156 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:03.935914 32241 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.929141 32260 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:03.929194 32216 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:03.940153 32216 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:03.928807 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:03.933990 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1021) took 1333 ms. Trace:
I20240426 13:22:03.942581 31461 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.943048 25105 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20240426 13:22:03.944401 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.953337 25100 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:22:03.945240 32246 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.203s	user 0.004s	sys 0.001s
I20240426 13:22:03.945858 31884 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.945791 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:03.950750 25615 rpcz_store.cc:276] 0426 13:22:02.600452 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:02.601024 (+   572us) service_pool.cc:225] Handling call
0426 13:22:02.601316 (+   292us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:03.763759 (+1162443us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:03.933766 (+170007us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540033a254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:03.933915 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2672066304}
I20240426 13:22:03.958612 25101 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:22:03.912464 31575 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:03.947326 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:03.953797 25618 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:22:03.956457 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.961089 25026 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:03.961791 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:03.962740 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.963878 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:03.966226 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.959765 25100 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:22:03.966619 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.966892 25030 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:03.967746 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:03.967902 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:03.967923 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:03.968683 25099 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:03.940778 32217 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:03.970631 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.973321 24955 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:03.961297 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.971128 24952 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:03.972311 25615 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:22:03.974915 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.976025 25100 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: []
I20240426 13:22:03.973052 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.977134 25616 raft_consensus.cc:2396] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:22:03.975672 25101 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:03.979936 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:03.981379 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.981495 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:03.982785 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.981936 24953 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:03.984469 24951 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:03.983305 24949 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:22:03.987114 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.988449 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.967968 31884 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:03.989015 24952 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:22:03.987811 25104 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:03.993388 24955 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:03.993007 32127 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:04.011379 32127 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.011871 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:04.012145 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:03.934515 31902 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:04.005707 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:03.990545 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.015897 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.016317 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.012548 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:03.991011 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:22:04.018404 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:04.021833 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.021998 31292 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:04.022552 31292 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.022966 31292 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:03.972188 32220 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:04.028152 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:04.028162 32075 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.030133 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.038210 25621 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:04.032052 25104 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:04.037446 24951 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:22:04.044663 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:04.032660 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:03.978013 32248 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.049990 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:03.957445 31461 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.050303 25100 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:04.066237 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.066743 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:04.013278 32110 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.088924 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:04.014444 32217 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:03.970255 32213 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
W20240426 13:22:04.126036 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 86840.7us
I20240426 13:22:03.990027 32261 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:22:04.090145 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.090440 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:04.066100 32075 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.050791 32220 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.122342 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:04.174748 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52492.8us
I20240426 13:22:03.972383 31575 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.114262 25100 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20240426 13:22:03.984771 32225 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20240426 13:22:04.121119 31702 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:04.018743 32060 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:04.020915 32127 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:04.124657 25104 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:22:04.100889 31292 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.188019 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:04.127723 25023 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:04.182122 25026 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:04.122898 32103 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:04.182683 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54780.9us
I20240426 13:22:04.208490 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:04.120311 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:04.208949 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 182500us
I20240426 13:22:04.031916 31930 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.142s
W20240426 13:22:04.185304 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94935.5us
I20240426 13:22:04.121482 31902 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:04.186956 32210 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:04.129076 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.217610 24951 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:04.126889 31751 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:04.179150 32198 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:04.210629 25098 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:04.213290 25096 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20240426 13:22:04.219663 25026 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:04.220136 24956 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:04.065671 32260 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20240426 13:22:04.121845 32174 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
W20240426 13:22:04.182816 32224 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:22:04.237223 32272 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:04.210409 32232 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.238128 32232 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.218381 32060 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.090224 32248 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.175123 32261 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:04.229665 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:04.246073 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62231.6us
I20240426 13:22:04.253407 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:04.254513 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:04.229928 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.255177 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:04.126335 32242 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20240426 13:22:04.184082 31721 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:04.188321 31702 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:04.260293 31702 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:04.234591 32256 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.235029 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.227900 32198 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.273924 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.180598 32169 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20240426 13:22:04.130776 32110 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:04.237576 32189 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.679s	user 0.007s	sys 0.000s
I20240426 13:22:04.306034 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:04.219267 32262 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
I20240426 13:22:04.222698 31292 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.187346 32226 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:04.211452 32216 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:04.128552 32162 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.320345 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:04.246892 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.222545 32210 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.221704 32078 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:04.266067 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:04.221501 32200 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:04.334966 32078 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:04.335460 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 69508.7us
I20240426 13:22:04.272267 24951 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
W20240426 13:22:04.302577 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48131.8us
I20240426 13:22:04.302672 32284 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:04.256456 31721 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.306689 24956 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:04.115437 32168 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.252619 32232 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.309024 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:04.310778 31292 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:04.219043 32103 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.311355 25097 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20240426 13:22:04.269188 32037 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.273119 31980 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:04.321348 25621 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:22:04.299121 32198 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:04.332927 25100 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:04.333948 32162 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.334492 32210 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:04.227849 31751 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.351821 31902 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:04.345201 24952 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 18: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:04.345989 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.396690 24949 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:04.350816 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.306318 32188 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:22:04.357350 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 136965us
I20240426 13:22:04.367447 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:04.300745 32218 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.335386 32078 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:04.242120 32217 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.382548 25024 raft_consensus.cc:1203] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Deduplicated request from leader. Original: 19.10->[20.11-20.11]   Dedup: 20.11->[]
I20240426 13:22:04.419042 25024 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20240426 13:22:04.356844 31721 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:04.385875 31292 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.420562 31292 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.335367 32200 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.381592 32232 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.320703 32076 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20240426 13:22:04.399907 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:04.401537 25104 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:04.402109 32156 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.360355 32260 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:22:04.388260 31859 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:04.405323 32266 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20240426 13:22:04.407371 32271 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:22:04.410230 25096 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:22:04.397900 24951 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:04.323215 32226 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.431856 31972 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.528854 25622 raft_consensus.cc:2354] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:04.369138 31461 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.423887 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:04.421836 25026 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:04.536726 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.430887 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.403522 32231 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:04.385512 31980 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:04.393429 31751 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:04.439569 25620 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
W20240426 13:22:04.404367 32273 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20240426 13:22:04.546958 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.414813 32162 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.389503 31884 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.384187 32261 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.485553 31575 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.361228 32026 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20240426 13:22:04.370740 32037 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.485334 32220 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.422555 31902 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:04.387310 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:04.560690 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 173496us
I20240426 13:22:04.561556 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:04.564219 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.564966 25621 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:04.424182 32188 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.502910 32210 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.504215 32257 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:04.504706 32276 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20240426 13:22:04.507911 32110 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.508440 32060 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.512712 32075 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.514420 25103 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20240426 13:22:04.516749 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:04.520593 25029 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:04.521178 32103 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.526289 32248 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.527480 32168 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.520017 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.532286 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:04.533131 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:04.548516 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.484413 32282 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:04.425788 32174 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:04.435971 32232 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:04.543108 32176 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.000s
I20240426 13:22:04.485105 31292 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:04.557296 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 219821us
W20240426 13:22:04.502698 32274 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20240426 13:22:04.541513 32231 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.573477 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.569924 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:04.660581 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127564us
I20240426 13:22:04.484920 31859 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:04.572244 32215 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.729s	user 0.010s	sys 0.000s
I20240426 13:22:04.508495 32286 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.101s
I20240426 13:22:04.638249 24948 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:22:04.639557 25030 raft_consensus.cc:2396] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:22:04.541110 31972 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.577049 32260 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:22:04.650849 25026 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:04.580235 32257 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.657008 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.707026 25621 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
W20240426 13:22:04.658123 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 125947us
I20240426 13:22:04.582582 32256 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.708187 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:04.576383 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:04.708797 25103 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:22:04.587392 32218 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.555507 32162 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.591542 32242 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:04.665711 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.664019 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.635993 31862 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:04.702781 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:04.727942 25022 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:04.728891 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:04.730268 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:04.663527 32231 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:04.704864 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.712462 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.713383 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.733379 25026 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:04.655997 31461 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:04.695583 32165 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.910s	user 0.010s	sys 0.000s
W20240426 13:22:04.718299 32246 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20240426 13:22:04.743744 24953 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:04.744087 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.716519 32257 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:04.746882 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:04.747462 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 208056us
I20240426 13:22:04.748332 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:04.751515 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.752012 24947 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:22:04.749042 25026 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:04.714444 32215 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:04.719707 31884 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:04.722838 32176 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:04.723078 32060 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:04.737991 32189 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:04.693399 32219 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.054s
I20240426 13:22:04.761679 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.778949 25026 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20240426 13:22:04.761982 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.760115 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:04.755405 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.687842 32282 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.764086 32156 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.767299 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.780231 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:04.768508 32226 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.768841 32220 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:04.663354 32174 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.759028 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:04.773303 24947 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:22:04.793057 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.793637 24947 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:22:04.738078 32256 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:04.687780 31980 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.794981 25588 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:04.795996 24956 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:04.802749 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:04.760785 31884 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:04.736088 32242 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.774049 32200 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.850049 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:04.855710 24947 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:04.859901 24947 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:04.869305 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:04.870059 25098 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:04.723692 31902 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.881705 24950 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:04.870138 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6277899576264e68aa43514e5b5c676e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:04.882633 24956 raft_consensus.cc:595] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:04.883074 24956 raft_consensus.cc:603] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:22:04.879061 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84146.2us
I20240426 13:22:04.785892 32288 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:04.851358 32127 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.632s	user 0.007s	sys 0.003s
W20240426 13:22:04.890579 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 157665us
W20240426 13:22:04.894503 32292 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20240426 13:22:04.895576 32288 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.897495 32288 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.898456 32162 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.900246 32037 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.891446 32210 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:04.901883 31702 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.641s	user 0.005s	sys 0.005s
I20240426 13:22:04.902314 31702 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.892867 32188 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.779026 32060 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.905721 32060 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.909509 32282 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.880820 32256 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.911666 32256 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.848520 31884 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:04.918112 31721 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.919742 25101 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19.
I20240426 13:22:04.920413 32127 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.916086 32231 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.922395 32127 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.922693 32231 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.923164 31859 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:04.924736 25105 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:04.926280 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:22:04.926853 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46979.3us
I20240426 13:22:04.930310 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:04.929581 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 225660us
I20240426 13:22:04.775530 32176 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:04.784080 32189 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:04.938328 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:04.934396 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.995853 25618 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:04.997036 25621 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:04.935966 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.933635 25022 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:04.936321 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:04.998059 25098 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:04.889884 32156 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Leader election won for term 12
I20240426 13:22:05.002521 24920 leader_election.cc:398] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:05.008173 24920 leader_election.cc:401] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:05.009470 24920 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:22:04.792179 32287 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.137s
W20240426 13:22:05.004925 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72933.1us
I20240426 13:22:05.006582 25030 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20240426 13:22:04.927078 32219 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
W20240426 13:22:05.012501 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54786.3us
I20240426 13:22:05.016875 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.017216 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.020414 25026 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:05.021436 25098 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:04.884845 32165 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.116s
I20240426 13:22:04.939330 32225 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:05.029059 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:04.929457 32194 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.025s	user 0.011s	sys 0.000s
I20240426 13:22:05.038578 31702 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.043870 32189 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.035002 32215 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.019s
I20240426 13:22:05.018616 32210 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:05.045439 32287 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:05.047223 32169 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:04.932844 32299 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:05.074189 32299 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.074836 31884 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.076360 25096 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:22:05.076418 32299 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.077826 32232 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:04.905721 32174 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.000306 32226 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:05.040170 31972 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.078797 32226 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:05.079568 32198 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.000s
I20240426 13:22:05.070156 32219 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:05.057955 32194 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.079993 32198 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.083303 32219 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.080935 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:05.070052 32287 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.086864 24956 raft_consensus.cc:595] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.087319 32194 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.087599 32287 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.081060 32169 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.080724 32226 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.081450 32295 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.025s
I20240426 13:22:05.090241 32169 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.090382 32216 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.090902 32216 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.092026 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.092778 25105 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:05.093698 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.094197 25105 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.094381 32176 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.095134 32225 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:05.096201 31575 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:05.096666 31575 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.096328 32225 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.097245 31575 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:05.097759 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:05.099097 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.100841 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.066498 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:05.024283 32306 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:05.071179 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.094226 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2895) took 1078 ms. Trace:
I20240426 13:22:05.073037 31721 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:04.929358 32078 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.096722 32231 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:05.103998 25026 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
W20240426 13:22:05.057137 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 286246us
I20240426 13:22:05.106410 25615 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:22:05.106339 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:05.108877 25622 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:22:05.114794 25621 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.115134 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.039810 32156 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:05.112387 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.112974 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.108662 24955 rpcz_store.cc:276] 0426 13:22:04.015874 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.039386 (+ 23512us) service_pool.cc:225] Handling call
0426 13:22:04.039824 (+   438us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:04.040021 (+   197us) raft_consensus.cc:1520] Early marking committed up to index 17
0426 13:22:04.102087 (+ 62066us) log.cc:842] Serialized 14 byte log entry
0426 13:22:05.093718 (+991631us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b18002eb9a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8ffa66ac 00007fda8d3cf20c 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb
0426 13:22:05.093895 (+   177us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:05.093942 (+    47us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:05.093990 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.094051 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:05.094148 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10096,"prepare.queue_time_us":82218,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"spinlock_wait_cycles":32348032,"thread_start_us":101015,"threads_started":1,"wal-append.queue_time_us":98064}
I20240426 13:22:05.116322 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:05.111171 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:05.057561 25618 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:05.118371 24920 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:22:05.120465 24920 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20240426 13:22:05.120538 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.114198 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.121305 25026 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:05.122491 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.123852 25620 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20240426 13:22:05.124977 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.125938 25023 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.125849 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.011643 31751 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.130894 24949 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:05.119589 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:05.131716 25624 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:05.108287 32198 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.135720 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.138559 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.141968 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:05.139463 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.142760 24955 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.135392 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.144189 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.140739 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.146715 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.147899 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.115141 31721 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:05.152529 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:05.120008 32231 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.153309 24947 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:05.119191 32288 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:05.153515 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.163823 25023 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:05.140018 25098 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:05.154414 24951 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20240426 13:22:05.154320 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 28.
I20240426 13:22:05.141216 31979 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:05.152779 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.167636 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.157243 24956 raft_consensus.cc:595] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.160177 25103 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 12.
I20240426 13:22:05.158071 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:05.125514 32304 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:05.168187 24956 raft_consensus.cc:603] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:05.167752 25022 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:05.141480 32220 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:05.141371 32169 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:05.156217 32188 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:05.176319 32269 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.994s	user 0.003s	sys 0.000s
I20240426 13:22:05.180428 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2641) took 1236 ms. Trace:
I20240426 13:22:05.152290 31980 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election won for term 23
W20240426 13:22:05.162667 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.766s (SENT) [suppressed 30 similar messages]
I20240426 13:22:05.183140 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.169847 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.181052 32220 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.152642 32174 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:05.152549 32210 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:22:05.187114 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.165448 32288 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:05.190587 32231 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.180773 25102 rpcz_store.cc:276] 0426 13:22:03.943555 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.954960 (+ 11405us) service_pool.cc:225] Handling call
0426 13:22:03.955219 (+   259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:03.955529 (+   310us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:03.955557 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:03.956020 (+   463us) log.cc:842] Serialized 26 byte log entry
0426 13:22:04.073580 (+117560us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:04.073641 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:04.073738 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:05.175762 (+1102024us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:05.180121 (+  4359us) raft_consensus.cc:1671] finished
0426 13:22:05.180145 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:05.180353 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2094,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":11976,"spinlock_wait_cycles":5788416,"thread_start_us":90961,"threads_started":1,"wal-append.queue_time_us":113302}
I20240426 13:22:05.192169 31980 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:05.182370 32026 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:05.225469 32257 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.225920 32026 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.226497 32026 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:05.185472 31721 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.227615 32257 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.200726 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.237210 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2654) took 1225 ms. Trace:
I20240426 13:22:05.237625 25099 rpcz_store.cc:276] 0426 13:22:04.012281 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.096593 (+ 84312us) service_pool.cc:225] Handling call
0426 13:22:04.169153 (+ 72560us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:04.261271 (+ 92118us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:04.261299 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:04.261337 (+    38us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:04.261366 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.237014 (+975648us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:05.237058 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:05.237161 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211406592}
I20240426 13:22:05.241402 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:05.242040 25618 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:05.193008 25105 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:05.114425 32306 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.245139 25105 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:05.246094 32306 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.206799 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.249470 32311 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:05.249773 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 1001) took 1304 ms. Trace:
I20240426 13:22:05.250424 25025 rpcz_store.cc:276] 0426 13:22:03.945323 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.947116 (+  1793us) service_pool.cc:225] Handling call
0426 13:22:05.249595 (+1302479us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:05.249696 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":589157760}
I20240426 13:22:05.181968 32242 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.252440 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.253036 25030 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.253082 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.253947 32231 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:05.254617 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.254485 32231 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.256441 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.256671 32231 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.215689 25622 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:22:05.171109 31972 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:05.257660 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.257894 31972 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.258191 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.258798 25025 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:05.152392 32201 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20240426 13:22:05.259063 31751 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:05.224737 31884 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.259759 31751 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:05.180773 32169 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
W20240426 13:22:05.260141 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59509.8us
I20240426 13:22:05.260540 32169 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:05.261051 32169 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.262223 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.262027 32169 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.263370 32215 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:05.263887 32215 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:05.264672 32215 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.261021 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:05.212352 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.009s	sys 0.024s
W20240426 13:22:05.269117 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.009s	sys 0.024s
I20240426 13:22:05.225762 32078 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.228317 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 3162) took 1214 ms. Trace:
I20240426 13:22:05.258302 25098 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:05.229092 31721 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:05.229614 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.198089 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.280295 25103 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 23
W20240426 13:22:05.280898 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82916.6us
I20240426 13:22:05.210407 32232 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:05.223641 32220 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.195477 25621 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:05.249846 32311 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.258291 31972 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.224910 31980 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.270326 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.271281 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.294915 25027 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.223907 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.271342 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.296108 25102 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:05.296499 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.296931 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.296531 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.297561 25102 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:05.270567 25030 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:05.298875 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.276604 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.276018 25101 rpcz_store.cc:276] 0426 13:22:04.014365 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.047484 (+ 33119us) service_pool.cc:225] Handling call
0426 13:22:05.228101 (+1180617us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:05.228232 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1633539968}
I20240426 13:22:05.164819 32176 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.060s
I20240426 13:22:05.207422 32217 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader election won for term 28
I20240426 13:22:05.300714 25098 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:22:05.300976 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.301482 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.302189 25098 raft_consensus.cc:2354] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:05.303011 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:05.284267 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 121619us
I20240426 13:22:05.260161 31751 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.286155 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.283860 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.287204 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.288280 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.221529 32288 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.292762 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.293538 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.288599 25618 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:05.212179 31702 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:22:05.231104 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:05.305892 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:05.304260 25025 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:05.285276 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:05.305685 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.307752 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:05.305640 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.277966 32303 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:05.296217 32311 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:05.309610 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.310173 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.324898 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.310256 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.310424 31292 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:05.324987 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:05.324599 25027 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:05.328670 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.322281 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.318616 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.332675 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.326336 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.292001 32306 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:05.315708 25103 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:05.335024 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.335346 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:05.294184 32231 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:05.330425 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:05.339440 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.334071 25023 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:05.340282 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:05.340828 25102 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.342542 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.283624 32225 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20240426 13:22:05.343587 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.340198 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:05.344816 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.321666 32277 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:22:05.346096 32277 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.346431 32162 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:05.347059 32162 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:05.346854 32231 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.347550 32162 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:05.348613 32297 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:05.348881 32231 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:05.349138 32297 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:05.297333 32232 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:05.350705 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.351164 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.348466 32162 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.344985 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.309126 32194 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:05.355419 25027 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:05.358690 24996 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19.
I20240426 13:22:05.366303 24996 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:05.359422 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:05.323499 32303 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.327306 32147 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20240426 13:22:05.327704 32103 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:05.333300 32272 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:05.338477 32168 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:05.378239 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:05.219239 32156 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.410594 24956 raft_consensus.cc:595] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:05.364647 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:05.340955 31721 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:05.360545 32217 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:05.397477 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.408478 32322 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19.
I20240426 13:22:05.416952 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:05.343787 32174 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:05.411121 24956 raft_consensus.cc:603] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:05.417922 24921 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20.
I20240426 13:22:05.362526 32078 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:05.423461 24921 leader_election.cc:398] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:05.423808 24921 leader_election.cc:401] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:22:05.424741 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60203.1us
I20240426 13:22:05.425956 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.370497 32303 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.385273 32037 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:05.321194 32315 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20240426 13:22:05.434721 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.438376 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.446292 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.408591 32301 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.131s
I20240426 13:22:05.382469 32189 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20240426 13:22:05.457334 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.422279 32173 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.425226 32248 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:05.461218 25102 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:05.422530 32168 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:05.465364 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.467700 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:05.467739 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.430127 32103 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:05.387252 32216 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.442319 31721 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:05.434682 32174 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:05.476195 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.458715 32315 consensus_queue.cc:1225] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:22:05.483021 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.483884 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.484975 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:05.468576 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:05.449925 31461 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:05.463836 32248 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.516242 24956 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:22:05.462452 32169 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:05.473498 32168 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.516841 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:05.431473 32078 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.473841 32103 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:05.378782 32232 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.424124 31751 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.573745 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:05.547529 24955 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:05.478111 32110 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:05.700459 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:05.698457 25624 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.002s
I20240426 13:22:05.472213 32297 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.704885 25624 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 28.
W20240426 13:22:05.714466 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 198943us
I20240426 13:22:05.477854 32312 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:05.517630 32194 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:05.525471 32248 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:05.514984 31884 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:05.689776 31721 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.481144 32026 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.744309 25101 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20240426 13:22:05.523274 32257 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20240426 13:22:05.757699 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 241057us
I20240426 13:22:05.529934 32103 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:05.531128 32147 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:05.735342 24956 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:05.903367 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:05.718142 32304 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:05.772656 32271 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20240426 13:22:05.737396 31702 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:05.910130 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.929265 24954 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:05.523921 31461 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:05.716493 32299 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:05.932956 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:05.785414 32037 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:05.945021 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:05.949684 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2705) took 1431 ms. Trace:
I20240426 13:22:05.950166 25029 rpcz_store.cc:276] 0426 13:22:04.518022 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.785771 (+267749us) service_pool.cc:225] Handling call
0426 13:22:04.850244 (+ 64473us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:05.023071 (+172827us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:22:05.044936 (+ 21865us) log.cc:842] Serialized 14 byte log entry
0426 13:22:05.155057 (+110121us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:05.155136 (+    79us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:05.155192 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.949381 (+794189us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:05.949450 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:05.949610 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34673,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":97,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":5982,"spinlock_wait_cycles":396785536,"thread_start_us":109724,"threads_started":1,"wal-append.queue_time_us":139502}
I20240426 13:22:05.514757 32217 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.945086 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2644) took 1995 ms. Trace:
I20240426 13:22:05.958940 25101 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20240426 13:22:05.964921 25097 rpcz_store.cc:276] 0426 13:22:03.949685 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.950031 (+   346us) service_pool.cc:225] Handling call
0426 13:22:03.969306 (+ 19275us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:04.360576 (+391270us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.934381 (+1573805us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b540009d3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:05.944942 (+ 10561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":779840256}
I20240426 13:22:05.924264 32315 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:05.992538 25101 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:22:05.996696 25097 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
W20240426 13:22:05.984834 32188 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.808s	user 0.004s	sys 0.005s
I20240426 13:22:05.980396 31721 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:05.986804 32162 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:05.942356 31702 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:05.975281 32343 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:05.990672 32037 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:05.928576 32304 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.005111 32188 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:05.892791 32174 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.032449 24956 raft_consensus.cc:595] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.035110 24956 raft_consensus.cc:603] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:22:06.028843 25651 heartbeater.cc:644] Failed to heartbeat to 127.23.61.62:34153 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.61.62:34153 timed out after 15.000s (SENT)
W20240426 13:22:06.026173 32310 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20240426 13:22:06.035104 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82225.3us
I20240426 13:22:06.022074 32291 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:06.027925 32343 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:05.990864 32315 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.031899 32333 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:06.054332 32215 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.001s	sys 0.015s
I20240426 13:22:06.045742 32216 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:06.054142 31575 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.956s	user 0.008s	sys 0.004s
I20240426 13:22:06.030647 32037 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.056237 32343 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:06.058035 32078 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.087603 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 3171) took 1832 ms. Trace:
I20240426 13:22:06.089777 25101 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20240426 13:22:06.090190 25101 pending_rounds.cc:76] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 10
I20240426 13:22:06.090433 25101 pending_rounds.cc:97] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20240426 13:22:06.090685 25101 raft_consensus.cc:2866] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:06.081270 25624 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.11-10.11]
W20240426 13:22:06.073038 32215 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.002s	sys 0.015s
I20240426 13:22:06.088191 25100 rpcz_store.cc:276] 0426 13:22:04.254805 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.254945 (+   140us) service_pool.cc:225] Handling call
0426 13:22:06.087335 (+1832390us) spinlock_profiling.cc:243] Waited 77 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.087530 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176986496}
W20240426 13:22:06.100731 32311 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.783s	user 0.008s	sys 0.001s
I20240426 13:22:06.101195 32311 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.103255 32311 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.070884 32333 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:06.096783 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122258us
I20240426 13:22:06.105954 24955 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:06.074244 32026 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:06.145108 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 999) took 2533 ms (client timeout 1766 ms). Trace:
W20240426 13:22:06.112571 32326 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.553s	user 0.001s	sys 0.000s
W20240426 13:22:06.145704 25031 rpcz_store.cc:269] 0426 13:22:03.582951 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.583065 (+   114us) service_pool.cc:225] Handling call
0426 13:22:06.116485 (+2533420us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54001e3554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.116628 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1670279936}
W20240426 13:22:06.113884 32331 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
W20240426 13:22:06.145787 32328 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.437s	user 0.001s	sys 0.000s
I20240426 13:22:06.078919 31575 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.147215 31721 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.163699 32188 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.187781 25030 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 24: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:22:06.180408 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.029s	sys 0.041s
I20240426 13:22:06.188485 32329 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:22:06.189288 32348 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20240426 13:22:06.192840 32350 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20240426 13:22:06.193174 32335 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20240426 13:22:06.193591 32327 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
W20240426 13:22:06.193974 32332 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20240426 13:22:06.194506 31702 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.167174 31461 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.197899 32345 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20240426 13:22:06.199790 32304 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:06.203382 32334 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20240426 13:22:06.205273 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.208935 32287 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:06.209957 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3353) took 2249 ms (client timeout 1651 ms). Trace:
W20240426 13:22:06.212679 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2657) took 1458 ms (client timeout 1839 ms). Trace:
I20240426 13:22:06.228304 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3164) took 2213 ms. Trace:
I20240426 13:22:06.113775 32342 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.186s
W20240426 13:22:06.227537 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2892) took 2217 ms (client timeout 2019 ms). Trace:
I20240426 13:22:06.114701 32215 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20240426 13:22:06.224459 32360 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.233274 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.226136 32026 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:06.238299 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.029s	sys 0.041s
W20240426 13:22:06.231403 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.014s	sys 0.030s
W20240426 13:22:06.304669 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.015s	sys 0.030s
I20240426 13:22:06.274219 32345 consensus_queue.cc:1225] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 11.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20240426 13:22:06.308447 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 862190us
I20240426 13:22:06.310499 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.312901 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3175) took 1558 ms. Trace:
I20240426 13:22:06.315102 25025 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:06.313273 25105 rpcz_store.cc:276] 0426 13:22:04.754056 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.245883 (+491827us) service_pool.cc:225] Handling call
0426 13:22:05.246183 (+   300us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:05.246524 (+   341us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:05.246558 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:05.246969 (+   411us) log.cc:842] Serialized 26 byte log entry
0426 13:22:05.417860 (+170891us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:05.417900 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.417965 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:06.312691 (+894726us) raft_consensus.cc:1671] finished
0426 13:22:06.312709 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.312848 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":306,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":170539,"threads_started":1,"wal-append.queue_time_us":269861}
I20240426 13:22:06.180094 31721 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.235283 32188 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.250823 32329 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:06.251735 32303 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.801s	user 0.006s	sys 0.001s
W20240426 13:22:06.255553 31972 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.933s	user 0.009s	sys 0.000s
I20240426 13:22:06.240978 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1258) took 1100 ms. Trace:
I20240426 13:22:06.234835 32200 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
W20240426 13:22:06.262893 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.002s	sys 0.019s
W20240426 13:22:06.262147 32288 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.929s	user 0.006s	sys 0.004s
I20240426 13:22:06.281983 25104 rpcz_store.cc:276] 0426 13:22:04.014964 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.216502 (+201538us) service_pool.cc:225] Handling call
0426 13:22:04.216748 (+   246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:04.433079 (+216331us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.220942 (+1787863us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:06.228249 (+  7307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424002816}
W20240426 13:22:06.281113 24950 rpcz_store.cc:269] 0426 13:22:03.960460 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.960587 (+   127us) service_pool.cc:225] Handling call
0426 13:22:04.573291 (+612704us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:22:06.209776 (+1636485us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.209875 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510285952}
W20240426 13:22:06.281183 25096 rpcz_store.cc:269] 0426 13:22:04.754628 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.849775 (+ 95147us) service_pool.cc:225] Handling call
0426 13:22:06.212478 (+1362703us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.212617 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":517789440}
I20240426 13:22:06.329020 25105 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20240426 13:22:06.290356 32315 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.290755 32260 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:06.292001 31862 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.292322 32305 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.293499 32189 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.294477 32290 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.296312 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3179) took 1277 ms. Trace:
I20240426 13:22:06.296343 32103 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.298780 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.299528 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:06.285313 24952 rpcz_store.cc:269] 0426 13:22:04.010204 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.010329 (+   125us) service_pool.cc:225] Handling call
0426 13:22:06.221293 (+2210964us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.227474 (+  6181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":750303616}
I20240426 13:22:06.300618 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.248572 32333 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.268829 32248 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:06.327765 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.002s	sys 0.019s
I20240426 13:22:06.322945 25098 rpcz_store.cc:276] 0426 13:22:05.141004 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.344750 (+203746us) service_pool.cc:225] Handling call
0426 13:22:05.345006 (+   256us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:05.379393 (+ 34387us) raft_consensus.cc:1520] Early marking committed up to index 17
0426 13:22:05.379429 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:05.379877 (+   448us) log.cc:842] Serialized 26 byte log entry
0426 13:22:05.419375 (+ 39498us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:05.419423 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.419485 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:06.240746 (+821261us) raft_consensus.cc:1671] finished
0426 13:22:06.240765 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.240919 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34010,"raft.queue_time_us":34846,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"thread_start_us":32369,"threads_started":1,"wal-append.queue_time_us":80165}
I20240426 13:22:06.333066 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:06.333837 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:06.354315 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.328492 32200 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:06.297972 25100 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:06.306186 32360 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.327390 31575 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.336589 25103 rpcz_store.cc:276] 0426 13:22:05.019399 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.341741 (+322342us) service_pool.cc:225] Handling call
0426 13:22:05.374290 (+ 32549us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:05.409218 (+ 34928us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:05.409260 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:05.416830 (+  7570us) log.cc:842] Serialized 26 byte log entry
0426 13:22:05.516795 (+ 99965us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:05.516844 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:05.516912 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:06.296094 (+779182us) raft_consensus.cc:1671] finished
0426 13:22:06.296114 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.296259 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34510,"raft.queue_time_us":34985,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":11123,"thread_start_us":52188,"threads_started":1,"wal-append.queue_time_us":495872}
I20240426 13:22:06.352169 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.308161 32026 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.354895 25029 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:06.371707 32213 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:06.339900 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.375079 24954 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:22:06.356467 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 782786us
I20240426 13:22:06.330374 32304 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.380523 32304 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.383244 32304 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.389070 25029 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:06.332357 32260 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:06.330520 31972 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.332659 32288 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.329459 32303 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.444638 25104 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:22:06.394893 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.393949 24954 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:06.410822 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:06.426878 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:06.425526 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.433470 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.454270 25621 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:06.351765 32188 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.454206 25105 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:06.425899 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.426363 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.427301 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.460925 25023 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:06.411821 24950 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:06.462994 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.461027 25098 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:06.461560 25027 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:06.465140 25023 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:06.467391 25105 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:06.427713 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.459905 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.462639 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.458889 25096 raft_consensus.cc:2354] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
W20240426 13:22:06.452062 32173 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.988s	user 0.001s	sys 0.010s
W20240426 13:22:06.468580 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 97140.4us
I20240426 13:22:06.471205 25099 raft_consensus.cc:2354] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
W20240426 13:22:06.471159 32362 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20240426 13:22:06.445578 32200 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.336426 32305 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.473843 31972 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.352711 32290 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.474720 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.475342 24954 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:06.475682 32290 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.476306 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.477458 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:06.480549 32289 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 1.595s	user 0.009s	sys 0.011s
I20240426 13:22:06.480777 32103 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.419963 32333 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.481727 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1038) took 2225 ms. Trace:
I20240426 13:22:06.482321 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 2720) took 1184 ms. Trace:
I20240426 13:22:06.482852 25030 rpcz_store.cc:276] 0426 13:22:05.298278 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.298620 (+   342us) service_pool.cc:225] Handling call
0426 13:22:06.482156 (+1183536us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.482274 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2042121600}
I20240426 13:22:06.352123 32329 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.469381 25024 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:06.473865 32305 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.478787 32303 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.504480 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.505833 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.507062 32295 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:06.508227 32295 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.444835 32365 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.509068 32365 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.508970 32295 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.510232 24952 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:06.511253 32365 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.511317 32295 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.444957 32363 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.433884 32078 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:06.494442 32360 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.445127 32341 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.139s
I20240426 13:22:06.395085 31859 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20240426 13:22:06.477254 32260 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.482107 25616 rpcz_store.cc:276] 0426 13:22:04.255913 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.256577 (+   664us) service_pool.cc:225] Handling call
0426 13:22:04.300061 (+ 43484us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:04.307298 (+  7237us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:04.307328 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:04.313075 (+  5747us) log.cc:842] Serialized 26 byte log entry
0426 13:22:04.639995 (+326920us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:04.640050 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:04.640121 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:06.479888 (+1839767us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:06.481449 (+  1561us) raft_consensus.cc:1671] finished
0426 13:22:06.481474 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.481671 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6900,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13451,"spinlock_wait_cycles":3954560,"thread_start_us":206727,"threads_started":1,"wal-append.queue_time_us":395350}
I20240426 13:22:06.336311 31862 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.338953 32189 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.498756 31721 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.521905 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:06.519383 24949 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2903) took 1855 ms (client timeout 2168 ms). Trace:
W20240426 13:22:06.542213 24949 rpcz_store.cc:269] 0426 13:22:04.664187 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.664518 (+   331us) service_pool.cc:225] Handling call
0426 13:22:06.519165 (+1854647us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.519309 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":905166592}
I20240426 13:22:06.512099 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.543530 24952 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:22:06.500078 32248 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.525456 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.433750 32165 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.368903 32147 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.157s
I20240426 13:22:06.541029 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:06.548821 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:06.547479 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 188437us
I20240426 13:22:06.490589 32173 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:06.515952 32026 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.520571 32241 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.546599 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.519505 32288 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.512024 32295 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:06.550858 25104 raft_consensus.cc:2354] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23.
I20240426 13:22:06.481727 32333 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.554047 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:06.560189 25025 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:06.560550 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.564568 24949 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:22:06.559367 32241 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.512379 32078 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.508308 32343 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:06.621484 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65719.7us
I20240426 13:22:06.512144 32363 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:06.550231 32165 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.567898 32260 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:06.576774 31972 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:22:06.559962 32295 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:06.615281 24952 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:06.620544 25104 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:06.625098 24996 leader_election.cc:398] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23.
I20240426 13:22:06.640300 24996 leader_election.cc:401] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20240426 13:22:06.641038 31575 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:06.633672 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:06.554879 32248 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.644587 32248 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.645234 32343 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.646677 32248 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.647078 32189 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.649240 32290 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:06.649749 32290 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:06.650543 32290 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.650995 32277 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.655516 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:06.657708 25616 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20240426 13:22:06.660706 32209 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:06.660848 32336 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:06.661152 32209 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:06.661679 32336 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:06.662113 32209 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.662797 32336 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.614837 31721 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.673156 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.674468 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.645236 32165 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.654065 32248 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:06.675628 32277 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:06.695005 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.696059 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2889) took 2702 ms. Trace:
I20240426 13:22:06.701370 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.696597 24953 rpcz_store.cc:276] 0426 13:22:03.988386 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.988517 (+   131us) service_pool.cc:225] Handling call
0426 13:22:03.994362 (+  5845us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:04.744803 (+750441us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.687298 (+1942495us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b5400078ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:06.690683 (+  3385us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1722651520}
I20240426 13:22:06.702934 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:06.699699 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3374) took 1347 ms. Trace:
I20240426 13:22:06.704236 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.704504 24952 rpcz_store.cc:276] 0426 13:22:05.352388 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.698792 (+1346404us) service_pool.cc:225] Handling call
0426 13:22:06.699118 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.699309 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.699345 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.699400 (+    55us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:06.699454 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.699518 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.699635 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:06.678732 31902 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.676721 32147 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:06.709380 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3381) took 1287 ms. Trace:
I20240426 13:22:06.708227 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.710076 24952 rpcz_store.cc:276] 0426 13:22:05.421845 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.708826 (+1286981us) service_pool.cc:225] Handling call
0426 13:22:06.709021 (+   195us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.709119 (+    98us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.709134 (+    15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.709170 (+    36us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:06.709199 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.709245 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.709323 (+    78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:06.692787 32209 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:06.674260 32231 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20240426 13:22:06.724248 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2668) took 1428 ms. Trace:
I20240426 13:22:06.724843 25102 rpcz_store.cc:276] 0426 13:22:05.295708 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.464319 (+168611us) service_pool.cc:225] Handling call
0426 13:22:05.464582 (+   263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:05.974595 (+510013us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:22:05.995141 (+ 20546us) log.cc:842] Serialized 14 byte log entry
0426 13:22:06.095241 (+100100us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.095334 (+    93us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:06.095398 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.723926 (+628528us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b540009c4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:06.723993 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.724178 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":21823,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":1528,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":1172409600,"thread_start_us":99618,"threads_started":1,"wal-append.queue_time_us":151840}
I20240426 13:22:06.726531 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.707903 32248 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.723337 32370 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.710170 32290 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:06.728519 32370 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.729035 32290 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.729100 32248 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.729856 32215 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.729636 32290 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.730310 32215 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.730738 32370 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.678350 32329 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20240426 13:22:06.731542 32290 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.732307 32363 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.734272 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.734890 25097 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:06.736058 32215 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.736793 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:06.738519 32304 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:06.738910 32304 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:06.739238 32304 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:06.739416 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.676751 31721 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.746912 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.748018 31721 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.707870 32333 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.749601 24921 leader_election.cc:398] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:22:06.749922 24921 leader_election.cc:401] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:06.750538 32382 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19.
I20240426 13:22:06.752194 32343 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:06.753046 32285 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:06.753592 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:06.745945 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.697408 32295 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.754256 25621 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:06.724272 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3380) took 1303 ms. Trace:
I20240426 13:22:06.755115 24953 rpcz_store.cc:276] 0426 13:22:05.421310 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.708560 (+1287250us) service_pool.cc:225] Handling call
0426 13:22:06.723746 (+ 15186us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.723928 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.723955 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.723996 (+    41us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:22:06.724051 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.724112 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.724223 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:06.755970 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:06.756526 24953 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:06.756992 32078 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.757082 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1347) took 1289 ms. Trace:
I20240426 13:22:06.758123 32241 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.757563 24953 rpcz_store.cc:276] 0426 13:22:05.468563 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.756084 (+1287521us) service_pool.cc:225] Handling call
0426 13:22:06.756360 (+   276us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.756878 (+   518us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.757037 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:06.760178 24921 leader_election.cc:398] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:06.760491 24921 leader_election.cc:401] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:06.761708 32037 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.762143 32282 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:06.762619 32282 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:06.762868 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.763299 31930 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:06.763998 32333 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:06.765494 32333 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:06.765789 32333 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:06.712358 31902 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.766525 32333 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.767669 32333 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:06.745128 31862 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.768752 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:06.745131 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.769309 25105 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:06.769582 25616 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:06.770728 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:06.771560 32189 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:06.771978 32189 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:06.772516 32189 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:06.772671 24949 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:06.772045 32282 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.776376 32343 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.776855 32343 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.777978 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.780226 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.780859 25027 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:22:06.781447 25105 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:06.768004 31902 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:06.771721 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3521) took 1691 ms (client timeout 1586 ms). Trace:
W20240426 13:22:06.789338 25024 rpcz_store.cc:269] 0426 13:22:05.080267 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.101738 (+ 21471us) service_pool.cc:225] Handling call
0426 13:22:06.771563 (+1669825us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003f7854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.771660 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3109294336}
I20240426 13:22:06.784461 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:06.787101 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.792853 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.793551 25623 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20240426 13:22:06.769420 31862 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.793540 25618 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:06.791900 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:06.843248 31862 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.795531 32343 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.796202 32127 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:06.844691 32127 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:06.845093 32127 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:06.845988 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.847064 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.763460 32245 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:22:06.801748 32304 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.803077 31575 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23.
I20240426 13:22:06.803594 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:06.795871 32156 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.009s
I20240426 13:22:06.808605 32189 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.866825 32245 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:06.868257 32189 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.813124 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.868939 25027 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:06.869490 32198 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:06.869880 32198 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:06.870222 32198 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:06.858498 32338 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.815476 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.873996 32338 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.874914 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.816432 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.817461 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.818789 25029 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:22:06.880914 25023 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:06.819247 32285 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:06.819443 32078 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:06.821731 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.890334 25027 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
W20240426 13:22:06.890543 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68900.8us
I20240426 13:22:06.891446 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.825567 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.827759 32315 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:06.837523 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.838011 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.838641 32290 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.838719 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1345) took 1757 ms. Trace:
W20240426 13:22:06.842363 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3383) took 1360 ms (client timeout 1767 ms). Trace:
W20240426 13:22:06.898479 24955 rpcz_store.cc:269] 0426 13:22:05.481864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.700111 (+218247us) service_pool.cc:225] Handling call
0426 13:22:06.842172 (+1142061us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b540038f4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.842287 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":930664832}
I20240426 13:22:06.854555 31902 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20240426 13:22:06.839675 25028 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3499) took 2888 ms (client timeout 1651 ms). Trace:
W20240426 13:22:06.855875 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 64047.4us
I20240426 13:22:06.880978 25026 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:06.822759 32312 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20240426 13:22:06.896286 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 70788.5us
W20240426 13:22:06.903827 25028 rpcz_store.cc:269] 0426 13:22:03.951708 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:03.952215 (+   507us) service_pool.cc:225] Handling call
0426 13:22:06.839481 (+2887266us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54002add54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.839607 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1823351168}
I20240426 13:22:06.794526 32276 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:06.897109 24948 rpcz_store.cc:276] 0426 13:22:05.081630 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.081801 (+   171us) service_pool.cc:225] Handling call
0426 13:22:05.082141 (+   340us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:05.095125 (+ 12984us) raft_consensus.cc:1520] Early marking committed up to index 18
0426 13:22:05.095394 (+   269us) log.cc:842] Serialized 14 byte log entry
0426 13:22:05.169046 (+ 73652us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:05.169160 (+   114us) raft_consensus.cc:1631] Marking committed up to 18
0426 13:22:05.169242 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.838434 (+1669192us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:06.838499 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:06.838653 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4405,"prepare.queue_time_us":4267,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":4574,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":30350,"spinlock_wait_cycles":29268864,"thread_start_us":68785,"threads_started":1,"wal-append.queue_time_us":98014,"wal-append.run_cpu_time_us":974,"wal-append.run_wall_time_us":1001021}
I20240426 13:22:06.709800 32147 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.900318 24955 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:22:06.910137 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.905129 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.912379 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:06.913163 25616 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 21
I20240426 13:22:06.919477 25023 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:06.921635 25030 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:06.875972 32338 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.926889 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:06.922420 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.902657 32295 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:06.895597 32241 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:06.923967 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:06.903132 32389 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20240426 13:22:06.866833 32127 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.903537 32299 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:06.922217 25616 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:06.904779 32156 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.929095 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:06.939699 32322 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:06.952266 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:06.952790 32322 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:06.953655 32322 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.944450 32285 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.945917 31862 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.908047 32290 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.960711 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:06.946363 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 238839us
I20240426 13:22:06.950407 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:06.954981 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:06.899366 32315 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.954635 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.926630 31575 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20240426 13:22:06.922959 32276 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:06.979712 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 2670) took 1675 ms (client timeout 1833 ms). Trace:
I20240426 13:22:06.980504 32276 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.981356 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.968312 25103 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20240426 13:22:06.972167 32315 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:06.976058 32147 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:06.976334 32225 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:06.957651 32156 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:06.992751 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:06.977322 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.982975 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:06.980162 25099 rpcz_store.cc:269] 0426 13:22:05.304742 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.322018 (+ 17276us) service_pool.cc:225] Handling call
0426 13:22:06.979536 (+1657518us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540001b3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.979656 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2620050816}
I20240426 13:22:06.980352 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:06.987951 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 29 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.999864 25621 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:07.000298 25623 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 29 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20240426 13:22:07.000226 32254 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:06.948319 32299 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:07.001291 32299 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:07.002195 32194 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:07.003304 32127 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:06.977217 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.005079 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.005125 31575 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20240426 13:22:06.993743 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:07.006705 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34068 (request call id 4) took 1515 ms. Trace:
I20240426 13:22:07.007505 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:06.998550 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:07.007513 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.006637 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.007838 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:06.984429 32322 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:06.996150 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:06.995400 25618 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 19. Current term is 21. Ops: []
I20240426 13:22:07.000869 32254 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.008412 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.044148 25023 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 20: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:07.046281 32254 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.047065 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.048357 25104 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:07.007234 24956 rpcz_store.cc:276] 0426 13:22:05.477189 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.932807 (+455618us) service_pool.cc:225] Handling call
0426 13:22:06.992163 (+1059356us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:06.992272 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207062784}
I20240426 13:22:07.049594 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.050086 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.051000 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.051272 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.055088 31884 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.058768 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.059664 24948 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19.
W20240426 13:22:07.059945 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52510.2us
I20240426 13:22:07.060636 32307 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:07.065511 32198 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:07.068238 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 75544.8us
I20240426 13:22:07.068900 32290 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.069272 31972 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:07.069810 31972 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:07.070667 31972 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:06.987434 32217 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.081825 32217 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.040252 32201 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:07.075889 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.084702 32201 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:07.085908 32201 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.055616 31884 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.076102 32303 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:07.076932 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:07.000792 32156 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.145934 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.003322 32194 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:07.094810 25097 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 12.
I20240426 13:22:07.153810 31292 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.154278 31292 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.157421 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.165903 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.120355 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.184191 25103 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20240426 13:22:07.008934 32315 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.145262 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:07.166751 25098 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:07.204890 25618 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20240426 13:22:07.205865 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:07.139875 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 141377us
I20240426 13:22:07.039351 32188 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:07.207154 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.211275 25098 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: [20.12-20.12]
W20240426 13:22:07.207252 32277 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.001s
W20240426 13:22:07.216289 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 220443us
I20240426 13:22:07.216972 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.217828 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.146135 32225 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:07.154757 32333 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:07.146651 32174 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.210325 32254 raft_consensus.cc:2741] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20240426 13:22:07.234223 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.235653 25615 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:22:07.215557 32282 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:07.224637 24947 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20240426 13:22:07.236349 24947 pending_rounds.cc:76] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:22:07.236617 24947 pending_rounds.cc:97] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20240426 13:22:07.236562 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.237046 24947 raft_consensus.cc:2866] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:07.226698 31292 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.227519 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.238831 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 29 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.155107 32209 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:07.239506 32209 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.207881 32276 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.207609 32156 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:07.244833 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.148123 32304 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.208958 32366 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:22:07.204952 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59927.2us
I20240426 13:22:07.242439 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.253576 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.242931 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.249235 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:22:07.246923 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:07.255467 25618 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:07.185073 32307 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.051s
I20240426 13:22:07.258399 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.226500 32188 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:07.230367 32333 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:07.265890 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:07.230304 32284 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.261886 25096 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:07.258397 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.242993 32241 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.264052 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:07.264554 24995 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19.
I20240426 13:22:07.184952 32194 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.219015 32171 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20240426 13:22:07.359225 25618 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:07.213896 32315 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.352510 32333 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:07.407423 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:07.354456 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:07.370363 25098 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:07.241726 32198 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.215402 32261 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20240426 13:22:07.246193 32189 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:07.254021 32156 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.252410 32364 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:07.258044 32218 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:07.375108 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.458412 25104 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:07.458876 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1346) took 2124 ms. Trace:
I20240426 13:22:07.459277 24947 rpcz_store.cc:276] 0426 13:22:05.334025 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.413855 (+ 79830us) service_pool.cc:225] Handling call
0426 13:22:05.414291 (+   436us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.237475 (+1823184us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.458686 (+221211us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:07.458827 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4162452736}
I20240426 13:22:07.460300 31884 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.460765 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.460980 32188 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.462407 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.378479 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:07.465909 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 87536.9us
W20240426 13:22:07.467628 32396 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20240426 13:22:07.469487 32198 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:07.383294 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 121417us
W20240426 13:22:07.365916 32385 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20240426 13:22:07.303113 32217 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:07.402354 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 230672us
I20240426 13:22:07.247383 31930 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:07.245271 32174 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.257290 32311 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.138s
I20240426 13:22:07.240737 32276 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.448110 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:07.507581 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 249234us
I20240426 13:22:07.374061 32336 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.383869 32346 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.263177 32254 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.146s
I20240426 13:22:07.456698 24956 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:07.402071 32225 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.263583 32227 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:07.471118 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:07.500641 32383 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20240426 13:22:07.463829 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:07.408738 32303 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:22:07.481087 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:07.490583 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.434075 32299 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.508385 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.460810 32171 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.401664 32329 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.261050 32366 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.403883 32241 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.540463 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.537763 25098 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:07.507345 32311 consensus_queue.cc:1225] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
W20240426 13:22:07.479409 32380 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20240426 13:22:07.560659 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.561755 24947 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:07.562747 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.389612 32284 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.554037 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:07.550395 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69417.3us
I20240426 13:22:07.553237 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:07.556615 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48314.2us
W20240426 13:22:07.553345 31859 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.014s	sys 0.011s
I20240426 13:22:07.585907 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:07.525168 32336 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.557293 31930 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:07.591275 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:07.560415 32217 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:07.597000 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.597529 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.598474 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:07.558004 32364 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:07.538866 32346 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:07.587716 31859 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.015s	sys 0.011s
I20240426 13:22:07.601112 24948 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
W20240426 13:22:07.608387 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.022s	sys 0.030s
I20240426 13:22:07.580505 32305 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:07.552570 32171 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:07.640220 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.022s	sys 0.030s
I20240426 13:22:07.642966 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1507) took 2711 ms. Trace:
I20240426 13:22:07.644176 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.595691 31979 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:07.646966 25023 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:22:07.647213 25104 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20240426 13:22:07.642858 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3382) took 2191 ms. Trace:
I20240426 13:22:07.643388 25620 rpcz_store.cc:276] 0426 13:22:04.931354 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:04.931980 (+   626us) service_pool.cc:225] Handling call
0426 13:22:05.123546 (+191566us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:05.127280 (+  3734us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.642789 (+2515509us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400491950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:07.642910 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284031232}
I20240426 13:22:07.648891 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.657290 24952 rpcz_store.cc:276] 0426 13:22:05.454317 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.711024 (+1256707us) service_pool.cc:225] Handling call
0426 13:22:06.711268 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.711803 (+   535us) raft_consensus.cc:1520] Early marking committed up to index 12
0426 13:22:06.904232 (+192429us) log.cc:842] Serialized 14 byte log entry
0426 13:22:07.642554 (+738322us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:07.642613 (+    59us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:07.642653 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.642700 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:07.642800 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":218963,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":81,"thread_start_us":930194,"threads_started":2}
I20240426 13:22:07.660352 24952 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:22:07.559860 32329 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:07.611532 32322 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.001s
I20240426 13:22:07.609256 32189 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:07.664161 25022 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:07.608319 32379 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:07.605134 32215 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:07.607960 31884 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:07.665503 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2916) took 1339 ms. Trace:
I20240426 13:22:07.689805 24948 rpcz_store.cc:276] 0426 13:22:06.325679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.640376 (+1314697us) service_pool.cc:225] Handling call
0426 13:22:07.664869 (+ 24493us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.665088 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:07.665127 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:07.665186 (+    59us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:07.665241 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.665310 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:07.665436 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:07.690761 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.666785 32194 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.666960 32156 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.658872 32305 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.667838 32171 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.668167 32312 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:07.672998 32174 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:07.676005 32399 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20240426 13:22:07.677248 32276 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.677606 32315 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.610927 32364 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.677913 32241 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.679065 32366 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:07.679416 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3384) took 1597 ms (client timeout 1651 ms). Trace:
I20240426 13:22:07.709618 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:07.709559 24950 rpcz_store.cc:269] 0426 13:22:06.081921 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.339093 (+257172us) service_pool.cc:225] Handling call
0426 13:22:07.679208 (+1340115us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:07.679350 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94653696}
I20240426 13:22:07.710546 32364 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.712323 32329 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.713807 32336 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:07.715020 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.717500 32284 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.706597 32407 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20240426 13:22:07.699052 31884 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:07.720621 31884 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.721521 32346 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.723289 32382 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:07.723793 32382 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:07.701300 32194 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20240426 13:22:07.724768 32382 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.726111 32382 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:07.727129 32295 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.794s	user 0.010s	sys 0.001s
I20240426 13:22:07.682150 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:07.644623 32394 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20240426 13:22:07.688258 25026 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20240426 13:22:07.610057 31930 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:07.603045 32232 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.062s
I20240426 13:22:07.661077 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2918) took 1333 ms. Trace:
I20240426 13:22:07.732859 24947 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20240426 13:22:07.694746 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:07.733079 24952 rpcz_store.cc:276] 0426 13:22:06.327301 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.659611 (+1332310us) service_pool.cc:225] Handling call
0426 13:22:07.660050 (+   439us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.660767 (+   717us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.661010 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:07.733738 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:07.605213 32218 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20240426 13:22:07.679589 32189 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:07.706185 32398 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20240426 13:22:07.679725 32213 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:07.704416 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.707022 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.731915 25031 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20240426 13:22:07.696730 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:07.734416 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:07.734473 25069 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.651s (SENT)
I20240426 13:22:07.736433 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2920) took 1382 ms. Trace:
I20240426 13:22:07.736692 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2919) took 1405 ms. Trace:
I20240426 13:22:07.695621 31721 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:07.702328 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.757071 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:07.746788 24947 rpcz_store.cc:276] 0426 13:22:06.331675 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.698408 (+1366733us) service_pool.cc:225] Handling call
0426 13:22:07.729373 (+ 30965us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:07.736463 (+  7090us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.736646 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:07.755749 24948 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:07.740072 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:07.746312 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.745972 24952 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:22:07.746341 24950 rpcz_store.cc:276] 0426 13:22:06.353956 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.710735 (+1356779us) service_pool.cc:225] Handling call
0426 13:22:07.711027 (+   292us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.736089 (+ 25062us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:07.736122 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:07.736165 (+    43us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:07.736194 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.736257 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:07.736382 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24743,"raft.queue_time_us":25558,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6193}
I20240426 13:22:07.762677 24952 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20240426 13:22:07.763165 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.763525 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:07.749490 31930 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.741303 32307 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.014s
I20240426 13:22:07.748777 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:07.736214 32299 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:07.741426 32189 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:07.767279 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2923) took 1400 ms. Trace:
I20240426 13:22:07.777038 24952 rpcz_store.cc:276] 0426 13:22:06.366886 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.762130 (+1395244us) service_pool.cc:225] Handling call
0426 13:22:07.762437 (+   307us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:07.767035 (+  4598us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.767228 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:22:07.768872 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65739.7us
I20240426 13:22:07.774801 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.782241 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.776913 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.786613 24950 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:07.779103 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1351) took 1345 ms. Trace:
I20240426 13:22:07.787417 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2924) took 1385 ms. Trace:
I20240426 13:22:07.787958 24950 rpcz_store.cc:276] 0426 13:22:06.402318 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.762143 (+1359825us) service_pool.cc:225] Handling call
0426 13:22:07.786421 (+ 24278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.787133 (+   712us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.787357 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:07.787801 24952 rpcz_store.cc:276] 0426 13:22:06.433380 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.778150 (+1344770us) service_pool.cc:225] Handling call
0426 13:22:07.778477 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.778687 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:07.778724 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:07.778783 (+    59us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:22:07.778853 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.778923 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:07.779040 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:07.795372 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.770035 32127 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:22:07.800503 32127 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:07.800962 32127 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:07.770323 32312 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:07.808176 32305 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.741223 32276 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20240426 13:22:07.811111 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.814685 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.787092 32322 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20240426 13:22:07.816352 32329 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:07.755024 32213 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.818756 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34690 (request call id 28) took 1003 ms. Trace:
I20240426 13:22:07.819442 32213 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:07.829607 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1517) took 1597 ms. Trace:
I20240426 13:22:07.830009 25624 rpcz_store.cc:276] 0426 13:22:06.232512 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.233939 (+  1427us) service_pool.cc:225] Handling call
0426 13:22:06.252310 (+ 18371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.419960 (+167650us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.419992 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.420051 (+    59us) raft_consensus.cc:1631] Marking committed up to 16
0426 13:22:06.420084 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:07.829390 (+1409306us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:07.829442 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:07.829556 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":385075072}
I20240426 13:22:07.771227 32232 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:07.827769 24947 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:07.833642 24947 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:07.842780 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:07.819120 25025 rpcz_store.cc:276] 0426 13:22:06.815294 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.849624 (+ 34330us) service_pool.cc:225] Handling call
0426 13:22:07.818291 (+968667us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:07.818667 (+   376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71833856}
I20240426 13:22:07.861456 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.861737 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.866683 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:22:07.867624 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77042.2us
I20240426 13:22:07.869782 24950 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:07.870397 24950 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:07.834549 24947 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.875114 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:07.795070 31859 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:07.759438 31721 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.880861 32254 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20240426 13:22:07.887610 32395 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.441s	user 0.000s	sys 0.001s
I20240426 13:22:07.908821 24948 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:22:07.890172 31751 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:07.898171 32379 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:07.876406 32307 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:07.881008 32336 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:07.920680 25617 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
W20240426 13:22:07.920796 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 134865us
I20240426 13:22:07.921897 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.924878 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.873405 24950 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.927044 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.895871 32189 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.928242 25027 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
W20240426 13:22:07.929955 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.000s	sys 0.013s
I20240426 13:22:07.931041 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:07.908335 32156 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.920258 32382 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:07.923933 32336 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:07.946714 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.929204 32226 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:07.929178 32299 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:07.974987 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.929562 32232 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.024s
I20240426 13:22:07.938903 32410 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20240426 13:22:07.977638 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.939203 32333 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:07.980808 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:07.979593 31884 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:07.976104 32382 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:07.984017 32382 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:07.965932 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.000s	sys 0.013s
I20240426 13:22:07.975311 32336 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:07.995816 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.003290 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:07.979590 32295 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:08.012985 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.008998 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2913) took 2675 ms. Trace:
I20240426 13:22:07.996287 32169 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:08.016312 24951 rpcz_store.cc:276] 0426 13:22:05.333743 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:05.381245 (+ 47502us) service_pool.cc:225] Handling call
0426 13:22:05.381556 (+   311us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:05.997727 (+616171us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:05.997788 (+    61us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:06.019086 (+ 21298us) log.cc:842] Serialized 52 byte log entry
0426 13:22:06.164819 (+145733us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:06.179635 (+ 14816us) log.cc:842] Serialized 14 byte log entry
0426 13:22:06.179922 (+   287us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.002362 (+1822440us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54001018d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.002417 (+    55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:08.002465 (+    48us) raft_consensus.cc:1671] finished
0426 13:22:08.008694 (+  6229us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.008914 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12761,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":261946,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":20605,"spinlock_wait_cycles":820295936,"thread_start_us":363526,"threads_started":2,"wal-append.queue_time_us":250594,"wal-append.run_cpu_time_us":18535,"wal-append.run_wall_time_us":1067161}
I20240426 13:22:08.021536 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.020872 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.001739 32410 consensus_queue.cc:1225] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20240426 13:22:08.023901 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1056) took 1253 ms. Trace:
I20240426 13:22:07.999817 32201 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20240426 13:22:08.034814 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1046) took 1706 ms. Trace:
I20240426 13:22:08.040086 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:08.039948 25622 rpcz_store.cc:276] 0426 13:22:06.328857 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.328978 (+   121us) service_pool.cc:225] Handling call
0426 13:22:06.757869 (+428891us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.758046 (+   177us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.758070 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.758111 (+    41us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:06.758141 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:06.758194 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.034643 (+1276449us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540058e650. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:08.034756 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":985715456}
I20240426 13:22:08.040913 24948 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:08.024502 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2675) took 1598 ms. Trace:
I20240426 13:22:08.036517 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:07.974231 32156 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.042562 25100 rpcz_store.cc:276] 0426 13:22:06.425658 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.426349 (+   691us) service_pool.cc:225] Handling call
0426 13:22:06.426626 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.654408 (+227782us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.654449 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.654504 (+    55us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:06.654556 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.024303 (+1369747us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.024347 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.024453 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":523258112}
I20240426 13:22:08.008183 32226 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:08.004997 32189 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:08.051648 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.039274 25619 rpcz_store.cc:276] 0426 13:22:06.770158 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.771096 (+   938us) service_pool.cc:225] Handling call
0426 13:22:06.771383 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.929713 (+158330us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.929743 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.929783 (+    40us) raft_consensus.cc:1631] Marking committed up to 18
0426 13:22:06.929814 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.023687 (+1093873us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.023730 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.023851 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":363721600}
I20240426 13:22:08.011744 32408 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:08.075492 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.079942 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:08.086390 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.087419 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.088598 25029 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:08.091473 24951 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:08.041620 32242 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:08.053830 32226 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.101248 25025 raft_consensus.cc:2396] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:22:08.101083 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.104635 25024 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:08.105638 25024 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:08.051613 32339 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.102640 25029 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:22:08.107407 24948 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:08.111780 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.113015 24948 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 24: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:08.115531 25029 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20240426 13:22:08.115995 25029 pending_rounds.cc:76] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 11
I20240426 13:22:08.116338 25029 pending_rounds.cc:97] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20240426 13:22:08.116645 25029 raft_consensus.cc:2866] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:08.103588 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1354) took 1116 ms. Trace:
I20240426 13:22:08.056834 32421 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:08.120308 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:08.088279 32232 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:08.121709 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.079422 32295 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:08.059161 32201 consensus_queue.cc:1225] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 17.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20240426 13:22:08.127733 24948 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:08.122514 24951 rpcz_store.cc:276] 0426 13:22:06.987691 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.058371 (+1070680us) service_pool.cc:225] Handling call
0426 13:22:08.090938 (+ 32567us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.102565 (+ 11627us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.103536 (+   971us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:08.128340 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1355) took 1136 ms. Trace:
I20240426 13:22:08.128778 24948 rpcz_store.cc:276] 0426 13:22:06.991690 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.114121 (+1122431us) service_pool.cc:225] Handling call
0426 13:22:08.127406 (+ 13285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.128120 (+   714us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.128292 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:08.129302 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:08.108688 32226 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:08.140314 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:08.108747 31721 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.142683 24951 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:08.127974 32305 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:08.150931 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3400) took 1002 ms. Trace:
I20240426 13:22:08.154589 24951 rpcz_store.cc:276] 0426 13:22:07.148852 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.140970 (+992118us) service_pool.cc:225] Handling call
0426 13:22:08.141321 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.149651 (+  8330us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.150880 (+  1229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:08.149865 32405 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
W20240426 13:22:08.157771 32413 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20240426 13:22:08.159756 32418 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20240426 13:22:08.163462 32419 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.123s	user 0.000s	sys 0.004s
W20240426 13:22:08.165082 32420 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.123s	user 0.002s	sys 0.003s
I20240426 13:22:08.166923 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2737) took 1313 ms. Trace:
I20240426 13:22:08.167379 25030 rpcz_store.cc:276] 0426 13:22:06.853227 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.875746 (+ 22519us) service_pool.cc:225] Handling call
0426 13:22:06.876001 (+   255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.921994 (+ 45993us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.166644 (+1244650us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.166847 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104204032}
I20240426 13:22:08.124504 32339 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.170337 32339 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.172513 32333 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:08.150362 32343 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:08.173272 32156 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.142288 32242 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:08.175892 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:08.176523 25618 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:08.139878 32299 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240426 13:22:08.176416 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.178402 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.179407 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.179478 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:08.151746 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131868us
I20240426 13:22:08.180967 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.173357 32343 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.182173 32343 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:08.182458 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:08.182838 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.183418 32209 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:08.186702 32417 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20240426 13:22:08.187446 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.188200 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2927) took 1280 ms. Trace:
W20240426 13:22:08.188371 32416 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20240426 13:22:08.188592 24956 rpcz_store.cc:276] 0426 13:22:06.907283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.111815 (+204532us) service_pool.cc:225] Handling call
0426 13:22:08.188046 (+1076231us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54001d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:08.188150 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":641318656}
I20240426 13:22:08.126968 32421 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.189649 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.190299 24956 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:08.190358 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:08.190821 24955 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:08.191071 32421 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.191457 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:08.192647 32330 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:08.193017 32330 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.193374 32330 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:08.195829 25031 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20240426 13:22:08.189419 25103 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:22:08.177665 32242 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.197335 32242 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:08.153573 32147 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20240426 13:22:08.203361 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.203577 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.203372 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3403) took 1053 ms. Trace:
I20240426 13:22:08.204386 24948 rpcz_store.cc:276] 0426 13:22:07.149582 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.154622 (+1005040us) service_pool.cc:225] Handling call
0426 13:22:08.202925 (+ 48303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.203095 (+   170us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.203120 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:08.203159 (+    39us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:08.203189 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.203234 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.203325 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:08.205502 24955 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
W20240426 13:22:08.208269 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53496us
I20240426 13:22:08.210497 25022 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:08.208787 25618 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:08.212194 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:08.213829 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.216197 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.214526 25028 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:08.220607 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.223393 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.225033 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.226094 25098 raft_consensus.cc:2396] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:22:08.226783 25618 raft_consensus.cc:1203] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Deduplicated request from leader. Original: 22.15->[23.16-28.18]   Dedup: 23.16->[26.17-28.18]
I20240426 13:22:08.210647 32110 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.227586 24948 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:08.227974 32110 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.228637 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.229430 24948 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:08.229271 32333 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:08.230608 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.231123 32232 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
I20240426 13:22:08.231571 24948 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20240426 13:22:08.231830 32225 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:08.155620 32366 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.084s
I20240426 13:22:08.231701 32330 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.222676 31862 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:08.237108 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1349) took 1998 ms. Trace:
I20240426 13:22:08.245481 24949 rpcz_store.cc:276] 0426 13:22:06.238841 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.831086 (+592245us) service_pool.cc:225] Handling call
0426 13:22:06.831410 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:06.972637 (+141227us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:06.972669 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:06.972710 (+    41us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:22:06.972740 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.236877 (+1264137us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.236930 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.237053 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":324363648}
I20240426 13:22:08.224066 25096 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:08.237761 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.166174 32336 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.241252 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:08.244531 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:08.258977 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.199687 32127 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.227591 25618 pending_rounds.cc:76] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 16
I20240426 13:22:08.280293 25618 pending_rounds.cc:97] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20240426 13:22:08.280683 25618 raft_consensus.cc:2866] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:08.285177 24949 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:08.286760 25619 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:08.232143 32423 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:08.292870 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.326506 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2928) took 1419 ms. Trace:
I20240426 13:22:08.326885 24952 rpcz_store.cc:276] 0426 13:22:06.907536 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.852309 (+944773us) service_pool.cc:225] Handling call
0426 13:22:07.854175 (+  1866us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:07.910729 (+ 56554us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:07.910776 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:07.922243 (+ 11467us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.104192 (+181949us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:08.291954 (+187762us) log.cc:842] Serialized 14 byte log entry
0426 13:22:08.326211 (+ 34257us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.326283 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:08.326325 (+    42us) raft_consensus.cc:1671] finished
0426 13:22:08.326335 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.326448 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80488,"prepare.queue_time_us":187773,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":56497,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":11910,"thread_start_us":294975,"threads_started":2,"wal-append.queue_time_us":207666}
I20240426 13:22:08.328011 31292 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:08.329727 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.318028 24950 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:08.322011 25617 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:22:08.322901 25620 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:08.254473 32353 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.237938 32200 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:08.324714 25100 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 23: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:08.293535 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 2696) took 1272 ms. Trace:
I20240426 13:22:08.340576 25102 rpcz_store.cc:276] 0426 13:22:07.020864 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.095940 (+ 75076us) service_pool.cc:225] Handling call
0426 13:22:07.150460 (+ 54520us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:07.449320 (+298860us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:07.449365 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:07.449424 (+    59us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:07.449468 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.293235 (+843767us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.293306 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.293465 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":686716160}
I20240426 13:22:08.252102 31721 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:08.239861 32299 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election won for term 19
W20240426 13:22:08.323542 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78955.3us
I20240426 13:22:08.250650 32305 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:08.334399 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.242537 32219 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.097s
I20240426 13:22:08.325400 32423 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.340184 31862 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:08.242286 32291 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
I20240426 13:22:08.325105 32410 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:08.335181 24952 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20240426 13:22:08.337996 31292 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:08.372851 24952 pending_rounds.cc:76] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 13
I20240426 13:22:08.373131 24952 pending_rounds.cc:97] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20240426 13:22:08.373358 24952 raft_consensus.cc:2866] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:08.362727 25622 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20240426 13:22:08.333094 32415 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:08.338104 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.369426 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.345249 31721 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:08.333942 32302 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20240426 13:22:08.391587 24950 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20240426 13:22:08.333418 32290 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 19.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20240426 13:22:08.398262 24950 pending_rounds.cc:76] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 12
I20240426 13:22:08.400786 24950 pending_rounds.cc:97] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20240426 13:22:08.401249 24950 raft_consensus.cc:2866] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:08.402195 31292 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:08.402526 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3395) took 1836 ms. Trace:
I20240426 13:22:08.402964 24950 rpcz_store.cc:276] 0426 13:22:06.566120 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.789688 (+1223568us) service_pool.cc:225] Handling call
0426 13:22:07.869626 (+ 79938us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.317093 (+447467us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b58008fc6b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:22:08.402255 (+ 85162us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.402477 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85096832}
I20240426 13:22:08.367164 32423 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:08.365774 32299 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:08.413959 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75925.7us
I20240426 13:22:08.342576 32353 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.374838 32225 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:08.444914 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.398322 32277 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20240426 13:22:08.374222 32260 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:22:08.501003 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86068.9us
W20240426 13:22:08.506165 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61381.1us
W20240426 13:22:08.509497 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106280us
I20240426 13:22:08.591661 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:08.560290 25103 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:07.757798 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:22:08.412443 32315 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20240426 13:22:08.412324 32340 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:08.446564 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47830.3us
I20240426 13:22:08.592567 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.596166 31292 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.601351 31292 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.448662 32285 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:22:08.446295 32304 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.404404 32231 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.187s
I20240426 13:22:08.622318 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3216) took 1400 ms. Trace:
I20240426 13:22:08.622856 25104 rpcz_store.cc:276] 0426 13:22:07.222011 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.459003 (+236992us) service_pool.cc:225] Handling call
0426 13:22:07.459348 (+   345us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:07.663141 (+203793us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.621968 (+958827us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54000ea454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:08.622242 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409237888}
I20240426 13:22:08.630435 25097 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:08.597296 32220 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.597937 32210 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:08.638104 31292 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:08.648656 32434 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20240426 13:22:08.651304 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1358) took 1404 ms. Trace:
I20240426 13:22:08.652426 31930 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.651739 24956 rpcz_store.cc:276] 0426 13:22:07.246765 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.190968 (+944203us) service_pool.cc:225] Handling call
0426 13:22:08.191293 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.191481 (+   188us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.191511 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.191923 (+   412us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.391059 (+199136us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:08.391083 (+    24us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.391156 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:08.651074 (+259918us) raft_consensus.cc:1671] finished
0426 13:22:08.651102 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.651245 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":145903,"threads_started":1,"wal-append.queue_time_us":416009}
I20240426 13:22:08.652935 31930 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.653371 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:08.653925 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:08.655117 31930 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:08.659143 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2935) took 1726 ms (client timeout 1704 ms). Trace:
W20240426 13:22:08.659546 24955 rpcz_store.cc:269] 0426 13:22:06.932426 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.951986 (+ 19560us) service_pool.cc:225] Handling call
0426 13:22:08.658992 (+1707006us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:08.659084 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2846549888}
I20240426 13:22:08.660609 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:08.661190 24955 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:08.666966 32218 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:08.655778 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2938) took 1425 ms. Trace:
I20240426 13:22:08.669931 24951 rpcz_store.cc:276] 0426 13:22:07.230780 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.155731 (+924951us) service_pool.cc:225] Handling call
0426 13:22:08.156116 (+   385us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.179608 (+ 23492us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.179657 (+    49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.180765 (+  1108us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.367891 (+187126us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:08.368295 (+   404us) log.cc:842] Serialized 14 byte log entry
0426 13:22:08.410569 (+ 42274us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.410664 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:08.655532 (+244868us) raft_consensus.cc:1671] finished
0426 13:22:08.655560 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:08.655712 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49329,"prepare.queue_time_us":1342,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":67795,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":35081,"thread_start_us":186849,"threads_started":1,"wal-append.queue_time_us":418308}
I20240426 13:22:08.671746 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:08.672075 24955 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:08.672377 24951 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
W20240426 13:22:08.673099 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72182.1us
I20240426 13:22:08.676616 24956 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:08.695937 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:08.699765 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.703532 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:08.696897 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:08.588644 32306 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20240426 13:22:08.610838 32301 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:08.739810 25097 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20240426 13:22:08.690757 32343 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.001s
I20240426 13:22:08.735245 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.741027 24955 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20240426 13:22:08.744014 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:08.716109 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:08.744866 24955 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20240426 13:22:08.745365 32423 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.741530 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:08.747381 32423 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:08.641499 32330 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.669286 32285 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:08.668589 32127 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:08.713752 32210 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:08.667423 32218 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:08.760062 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:08.765177 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49208.1us
I20240426 13:22:08.627821 32342 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.115s
I20240426 13:22:08.714593 32336 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:08.604813 31721 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.668052 32430 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20240426 13:22:08.608537 32305 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.646903 32299 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.749101 32261 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:08.621479 32304 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:08.782580 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 166675us
I20240426 13:22:08.782833 32217 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:08.785828 24947 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:08.797092 32110 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:08.799269 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 1535) took 1060 ms. Trace:
I20240426 13:22:08.841710 25620 rpcz_store.cc:276] 0426 13:22:07.738570 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.740687 (+  2117us) service_pool.cc:225] Handling call
0426 13:22:08.798970 (+1058283us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540063f454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:08.799164 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195900800}
I20240426 13:22:08.800009 25615 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:22:08.767699 24951 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:22:08.843446 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.766081 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:08.762653 32439 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.427s	user 0.001s	sys 0.000s
I20240426 13:22:08.692909 32220 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:08.832440 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 230805us
I20240426 13:22:08.847762 31721 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:08.760671 32406 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:08.843689 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:08.844262 24951 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 18.
I20240426 13:22:08.848593 25619 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:08.849263 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.848215 32220 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:08.827039 32194 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20240426 13:22:08.845757 32226 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.693s	user 0.009s	sys 0.000s
I20240426 13:22:08.848249 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:08.855567 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89571.6us
I20240426 13:22:08.762017 32286 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
I20240426 13:22:08.750782 32343 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:08.858667 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 161893us
I20240426 13:22:08.848270 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:08.763828 32213 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.080s
I20240426 13:22:08.829941 32342 consensus_queue.cc:1225] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:22:08.838560 31751 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.883949 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.871594 32301 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:08.890858 32301 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:08.891176 32301 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:08.857542 32220 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:08.880910 32217 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:08.892386 32110 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:08.883451 32218 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.894170 32218 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:08.892931 32217 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.892936 32110 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.767407 32210 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.855342 32261 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:08.855082 32194 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:22:08.916602 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.009s	sys 0.018s
W20240426 13:22:08.918437 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 153723us
W20240426 13:22:08.917121 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.009s	sys 0.018s
I20240426 13:22:08.944416 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.944622 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20240426 13:22:08.945008 24947 pending_rounds.cc:76] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 11
I20240426 13:22:08.945336 24947 pending_rounds.cc:97] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20240426 13:22:08.904711 32415 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:22:08.949494 24947 raft_consensus.cc:2866] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:08.947605 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:08.955242 25023 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:08.857467 32226 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.952278 32261 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:08.953444 32210 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:08.953824 32304 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:08.953732 32301 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.916095 32194 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:08.955466 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3396) took 1992 ms. Trace:
I20240426 13:22:08.898265 31751 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:08.857738 32406 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:08.961758 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 215719us
W20240426 13:22:08.955948 32339 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.015s	sys 0.015s
I20240426 13:22:08.930397 32353 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:08.978055 24947 rpcz_store.cc:276] 0426 13:22:06.962654 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:07.827396 (+864742us) service_pool.cc:225] Handling call
0426 13:22:07.827667 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.785260 (+957593us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b58004775b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:22:08.950090 (+164830us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.955415 (+  5325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":109706112}
I20240426 13:22:08.949360 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.993713 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:08.978773 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 208794us
W20240426 13:22:08.961131 32200 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.619s	user 0.008s	sys 0.001s
I20240426 13:22:08.995725 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:08.978363 32443 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20240426 13:22:08.998638 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:08.998546 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:08.999392 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:09.000546 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [16.13-18.14]
I20240426 13:22:08.997048 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:08.998834 24948 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20240426 13:22:08.977638 32245 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
W20240426 13:22:08.992624 32339 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.015s	sys 0.015s
I20240426 13:22:09.003728 24951 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:09.011008 24954 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20240426 13:22:08.916108 32452 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.089s
I20240426 13:22:09.020694 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:09.023331 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 180) took 18354 ms (client timeout 15000 ms). Trace:
I20240426 13:22:09.025238 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:09.026115 24947 raft_consensus.cc:2362] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21. Re-sending same reply.
I20240426 13:22:08.948379 32176 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20240426 13:22:09.002692 32147 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
W20240426 13:22:09.018857 32442 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20240426 13:22:09.021024 32312 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:09.056596 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:09.025241 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
W20240426 13:22:09.023880 24852 rpcz_store.cc:269] 0426 13:21:50.660283 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:50.660565 (+   282us) service_pool.cc:225] Handling call
0426 13:22:09.014134 (+18353569us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:01.537968 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:01.538986 (+  1018us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:01.539053 (+    67us) write_op.cc:427] Acquired schema lock
0426 13:22:01.539093 (+    40us) tablet.cc:660] Decoding operations
0426 13:22:01.541075 (+  1982us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:01.541176 (+   101us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:01.541218 (+    42us) tablet.cc:683] Acquiring locks for 15 operations
0426 13:22:01.616689 (+ 75471us) tablet.cc:699] Row locks acquired
0426 13:22:01.616729 (+    40us) write_op.cc:260] PREPARE: finished
0426 13:22:01.616898 (+   169us) write_op.cc:270] Start()
0426 13:22:01.704541 (+ 87643us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b54000047a8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda96845f71 00007fda96844bdc 00007fda93d3c899 00007fda93d27d53 00007fda93d27828 00007fda93d2c111 00007fda93d2c0c9 00007fda93d2c059 00007fda93d2c021 00007fda93d2b31d 00007fda96660d34
0426 13:22:01.704786 (+   245us) write_op.cc:275] Timestamp: P: 1714137721616829 usec, L: 0
0426 13:22:01.704846 (+    60us) op_driver.cc:331] REPLICATION: starting
0426 13:22:01.705537 (+   691us) log.cc:842] Serialized 5866 byte log entry
0426 13:22:01.913227 (+207690us) op_driver.cc:444] REPLICATION: finished
0426 13:22:02.064648 (+151421us) write_op.cc:300] APPLY: starting
0426 13:22:02.064797 (+   149us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:02.353319 (+288522us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:02.353372 (+    53us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:03.148497 (+795125us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:03.148906 (+   409us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0426 13:22:03.164665 (+ 15759us) write_op.cc:311] APPLY: finished
0426 13:22:03.329052 (+164387us) log.cc:842] Serialized 139 byte log entry
0426 13:22:03.364321 (+ 35269us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b1800357a00. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:22:03.376835 (+ 12514us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:03.377010 (+   175us) write_op.cc:446] Released schema lock
0426 13:22:03.377080 (+    70us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:03.377428 (+   348us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:03.377460 (+    32us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":120467,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":130455,"apply.run_cpu_time_us":29203,"apply.run_wall_time_us":1323771,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66857,"num_ops":15,"prepare.queue_time_us":64256,"prepare.run_cpu_time_us":33473,"prepare.run_wall_time_us":224560,"raft.queue_time_us":38357,"raft.run_cpu_time_us":7454,"raft.run_wall_time_us":20362,"replication_time_us":208266,"spinlock_wait_cycles":17006208,"thread_start_us":164677,"threads_started":4,"wal-append.queue_time_us":159377,"wal-append.run_cpu_time_us":9425,"wal-append.run_wall_time_us":2083332}]]}
W20240426 13:22:09.049526 32446 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20240426 13:22:09.032238 31930 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:09.061856 24947 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
W20240426 13:22:09.076373 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 127199us
I20240426 13:22:09.080078 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:09.082355 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:09.083377 25619 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19. Ops: [16.13-18.14]
I20240426 13:22:09.085459 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:09.088478 25619 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:09.069072 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:09.096804 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:09.059525 32147 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:22:09.069062 32444 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20240426 13:22:09.122354 24947 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:09.058387 32330 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:09.069447 31930 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:09.088892 32343 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:09.131294 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 134296us
I20240426 13:22:09.123713 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:09.089941 32357 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:09.084527 32281 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:09.128708 31902 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
W20240426 13:22:09.207051 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 123817us
W20240426 13:22:09.130378 32242 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.931s	user 0.011s	sys 0.002s
I20240426 13:22:09.169409 32336 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:09.089855 32465 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:09.068372 32312 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.216336 24947 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:22:09.216601 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:09.126794 32286 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:09.204192 32446 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20240426 13:22:09.221926 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:09.203764 31930 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
W20240426 13:22:09.234939 32378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.000s	sys 0.009s
I20240426 13:22:09.208289 32342 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.240331 25098 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:09.211315 32357 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:09.243649 32378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.000s	sys 0.009s
I20240426 13:22:09.244539 31751 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.218183 32110 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:09.257983 24947 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:09.204751 32330 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:09.400802 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146444us
I20240426 13:22:09.232007 31902 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:09.455168 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2952) took 1160 ms. Trace:
I20240426 13:22:09.456071 24947 rpcz_store.cc:276] 0426 13:22:08.240615 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.234032 (+993417us) service_pool.cc:225] Handling call
0426 13:22:09.257813 (+ 23781us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.398236 (+140423us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.400799 (+  2563us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:09.250227 32353 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:09.252228 32357 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:09.253099 32226 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.257114 32301 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.229885 32242 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.447098 32110 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:09.233390 32336 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:09.460948 32447 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.466s	user 0.000s	sys 0.001s
I20240426 13:22:09.461565 32353 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:09.237174 32286 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.252693 32342 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.486129 32406 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:09.476613 32209 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.490110 32315 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:09.532737 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 72161.1us
I20240426 13:22:09.517511 32353 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:09.264992 32147 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.536459 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:09.543346 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 1184) took 1326 ms. Trace:
I20240426 13:22:09.544410 25618 rpcz_store.cc:276] 0426 13:22:08.217488 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.226130 (+  8642us) service_pool.cc:225] Handling call
0426 13:22:08.226542 (+   412us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:22:08.281399 (+ 54857us) raft_consensus.cc:1520] Early marking committed up to index 16
0426 13:22:08.338108 (+ 56709us) log.cc:842] Serialized 14 byte log entry
0426 13:22:08.564409 (+226301us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:08.590443 (+ 26034us) log.cc:842] Serialized 52 byte log entry
0426 13:22:08.601546 (+ 11103us) raft_consensus.cc:1631] Marking committed up to 16
0426 13:22:08.601583 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.601642 (+    59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:08.788529 (+186887us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b48000b0658. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
0426 13:22:09.533985 (+745456us) raft_consensus.cc:1671] finished
0426 13:22:09.543096 (+  9111us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.543279 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50940,"prepare.queue_time_us":57386,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":38396,"raft.run_cpu_time_us":2656,"raft.run_wall_time_us":229842,"spinlock_wait_cycles":10439040,"thread_start_us":204108,"threads_started":1,"wal-append.queue_time_us":405177}
I20240426 13:22:09.561214 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:09.506652 32226 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20240426 13:22:09.511338 32110 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:09.552716 32458 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.486s	user 0.001s	sys 0.000s
I20240426 13:22:09.536015 32315 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:09.585733 32464 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.460s	user 0.000s	sys 0.001s
I20240426 13:22:09.536289 32209 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.569046 32295 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:09.590577 32463 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.419s	user 0.001s	sys 0.000s
W20240426 13:22:09.591312 32453 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.300s	user 0.000s	sys 0.001s
W20240426 13:22:09.614719 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 372342us
I20240426 13:22:09.612171 32168 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:09.621069 32403 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:09.623657 32168 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.563488 31902 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.653903 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1367) took 1413 ms. Trace:
I20240426 13:22:09.684530 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:09.649865 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2953) took 1409 ms. Trace:
I20240426 13:22:09.662185 24953 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20240426 13:22:09.673676 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:09.693702 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:09.691426 24955 rpcz_store.cc:276] 0426 13:22:08.240045 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.746409 (+506364us) service_pool.cc:225] Handling call
0426 13:22:08.746679 (+   270us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.749872 (+  3193us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.749927 (+    55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.760360 (+ 10433us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.960438 (+200078us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:08.993188 (+ 32750us) log.cc:842] Serialized 14 byte log entry
0426 13:22:09.010037 (+ 16849us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.010125 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:09.653719 (+643594us) raft_consensus.cc:1671] finished
0426 13:22:09.653739 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.653851 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20315,"prepare.queue_time_us":22685,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":9864,"raft.queue_time_us":20701,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":17137,"spinlock_wait_cycles":10689920,"thread_start_us":199476,"threads_started":1,"wal-append.queue_time_us":305539}
I20240426 13:22:09.694430 25025 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:09.672109 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:09.671739 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:09.624332 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:09.693189 24953 pending_rounds.cc:76] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:22:09.657620 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34690 (request call id 35) took 1016 ms. Trace:
W20240426 13:22:09.661805 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 3390) took 3308 ms (client timeout 2393 ms). Trace:
I20240426 13:22:09.697698 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:09.691757 24947 rpcz_store.cc:276] 0426 13:22:08.240881 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.456890 (+1216009us) service_pool.cc:225] Handling call
0426 13:22:09.477209 (+ 20319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.477665 (+   456us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:22:09.478236 (+   571us) log.cc:842] Serialized 14 byte log entry
0426 13:22:09.649441 (+171205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.649518 (+    77us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:09.649583 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.649666 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.649807 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33068,"prepare.queue_time_us":8920,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":17258,"raft.queue_time_us":35631,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":137559,"threads_started":1}
I20240426 13:22:09.652928 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1069) took 1241 ms. Trace:
I20240426 13:22:09.698664 25028 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:09.697270 24953 pending_rounds.cc:97] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20240426 13:22:09.699281 25624 rpcz_store.cc:276] 0426 13:22:08.411556 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.411728 (+   172us) service_pool.cc:225] Handling call
0426 13:22:08.508777 (+ 97049us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.509207 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.509248 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.599518 (+ 90270us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.858885 (+259367us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:22:08.858937 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.859026 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:09.652717 (+793691us) raft_consensus.cc:1671] finished
0426 13:22:09.652738 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.652875 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1265,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":32886,"thread_start_us":213980,"threads_started":1,"wal-append.queue_time_us":373914}
I20240426 13:22:09.649174 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3412) took 1569 ms. Trace:
I20240426 13:22:09.702589 24953 raft_consensus.cc:2866] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:09.699795 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:09.696949 25098 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 19
I20240426 13:22:09.704286 25024 raft_consensus.cc:2354] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:22:09.702049 24950 rpcz_store.cc:276] 0426 13:22:08.079849 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.500475 (+420626us) service_pool.cc:225] Handling call
0426 13:22:08.595466 (+ 94991us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:08.832220 (+236754us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.837868 (+  5648us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:08.838396 (+   528us) log.cc:842] Serialized 52 byte log entry
0426 13:22:09.219243 (+380847us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:09.219405 (+   162us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.219485 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:09.648871 (+429386us) raft_consensus.cc:1671] finished
0426 13:22:09.648903 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.649105 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11302,"raft.queue_time_us":38873,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":45607,"thread_start_us":604795,"threads_started":2,"wal-append.queue_time_us":715805}
W20240426 13:22:09.698483 24954 rpcz_store.cc:269] 0426 13:22:06.353173 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.477242 (+124069us) service_pool.cc:225] Handling call
0426 13:22:09.661576 (+3184334us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54000be854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:09.661734 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5804230912}
I20240426 13:22:09.697424 25031 rpcz_store.cc:276] 0426 13:22:08.641544 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.641684 (+   140us) service_pool.cc:225] Handling call
0426 13:22:09.657133 (+1015449us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:09.657560 (+   427us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230437248}
W20240426 13:22:09.633653 32471 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.167s	user 0.002s	sys 0.000s
I20240426 13:22:09.611933 32482 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:22:09.635959 32194 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.632s	user 0.006s	sys 0.001s
W20240426 13:22:09.661614 31721 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.813s	user 0.010s	sys 0.000s
W20240426 13:22:09.637346 32468 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.388s	user 0.001s	sys 0.000s
W20240426 13:22:09.631635 32476 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20240426 13:22:09.631171 32218 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.737s	user 0.007s	sys 0.005s
I20240426 13:22:09.743404 25025 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:09.739964 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2948) took 1827 ms. Trace:
I20240426 13:22:09.707096 32210 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:09.628903 32342 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:09.732669 32231 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:09.645347 32209 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.747874 32315 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.732728 32213 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:09.752839 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3419) took 1150 ms. Trace:
I20240426 13:22:09.629182 32110 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.623668 32242 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.701687 32201 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:09.757195 24952 rpcz_store.cc:276] 0426 13:22:07.912828 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.391392 (+478564us) service_pool.cc:225] Handling call
0426 13:22:08.391705 (+   313us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.391853 (+   148us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.391869 (+    16us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.392235 (+   366us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.734505 (+342270us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:08.761721 (+ 27216us) log.cc:842] Serialized 14 byte log entry
0426 13:22:08.761953 (+   232us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.762023 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:09.739663 (+977640us) raft_consensus.cc:1671] finished
0426 13:22:09.739691 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.739896 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27498,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":117,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":1277,"thread_start_us":288992,"threads_started":1,"wal-append.queue_time_us":524163}
I20240426 13:22:09.826668 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1371) took 1136 ms. Trace:
I20240426 13:22:09.822860 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:09.636078 32168 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.700959 32303 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:09.765435 25098 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:09.797104 25622 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:09.622989 32295 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.646219 32336 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:09.822465 24947 rpcz_store.cc:276] 0426 13:22:08.602655 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.699203 (+1096548us) service_pool.cc:225] Handling call
0426 13:22:09.752417 (+ 53214us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.752558 (+   141us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:09.752581 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.752617 (+    36us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:09.752647 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.752695 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.752790 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:09.827219 24952 rpcz_store.cc:276] 0426 13:22:08.689847 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.825710 (+1135863us) service_pool.cc:225] Handling call
0426 13:22:09.826098 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.826262 (+   164us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:09.826286 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.826327 (+    41us) raft_consensus.cc:1631] Marking committed up to 14
0426 13:22:09.826360 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.826421 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.826602 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:22:09.856242 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 244044us
I20240426 13:22:09.857211 24952 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20240426 13:22:09.857789 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2958) took 1005 ms. Trace:
I20240426 13:22:09.830729 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:09.858325 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:09.858925 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:09.859467 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.003s	sys 0.007s
I20240426 13:22:09.859740 32315 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20240426 13:22:09.859766 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.003s	sys 0.007s
I20240426 13:22:09.860224 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:09.623754 32403 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.864073 24952 rpcz_store.cc:276] 0426 13:22:08.852298 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.856605 (+1004307us) service_pool.cc:225] Handling call
0426 13:22:09.856915 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.857555 (+   640us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.857742 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:22:09.864993 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.005s	sys 0.005s
W20240426 13:22:09.825325 32305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.001s	sys 0.008s
W20240426 13:22:09.862289 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.003s	sys 0.007s
I20240426 13:22:09.754267 32194 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.824625 32286 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:09.889153 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2767) took 1004 ms. Trace:
I20240426 13:22:09.893946 25027 rpcz_store.cc:276] 0426 13:22:08.885133 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.885319 (+   186us) service_pool.cc:225] Handling call
0426 13:22:08.885674 (+   355us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.885931 (+   257us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.885968 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.895252 (+  9284us) log.cc:842] Serialized 26 byte log entry
0426 13:22:09.020815 (+125563us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:09.020848 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.020913 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:09.888950 (+868037us) raft_consensus.cc:1671] finished
0426 13:22:09.888974 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.889101 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":91351,"threads_started":1,"wal-append.queue_time_us":225842}
I20240426 13:22:09.888429 24947 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:09.860574 32315 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20240426 13:22:09.885156 32305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.001s	sys 0.008s
I20240426 13:22:09.904059 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2956) took 1066 ms. Trace:
I20240426 13:22:09.918876 24947 rpcz_store.cc:276] 0426 13:22:08.838097 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.856237 (+1018140us) service_pool.cc:225] Handling call
0426 13:22:09.888222 (+ 31985us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.903705 (+ 15483us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.903970 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:22:09.833951 32201 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.894174 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:09.808111 32210 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.751289 32482 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.807066 32357 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.823817 32231 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.887490 32110 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:09.932224 32168 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.922760 32218 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:09.930692 31930 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.678s	user 0.009s	sys 0.000s
I20240426 13:22:09.935551 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1370) took 1267 ms. Trace:
I20240426 13:22:09.936622 32194 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:09.938637 32209 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.939379 32423 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:09.885449 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.003s	sys 0.011s
W20240426 13:22:09.966907 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 855548us
I20240426 13:22:09.940057 32347 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:09.941290 32477 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.353s	user 0.000s	sys 0.001s
I20240426 13:22:09.941358 32496 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:09.829993 32213 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.887256 24952 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: []
W20240426 13:22:09.865374 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.005s	sys 0.005s
W20240426 13:22:09.947072 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84271.5us
I20240426 13:22:09.856215 32242 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:09.946107 24954 rpcz_store.cc:276] 0426 13:22:08.668136 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.710612 (+1042476us) service_pool.cc:225] Handling call
0426 13:22:09.710928 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.711154 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:22:09.757753 (+ 46599us) log.cc:842] Serialized 14 byte log entry
0426 13:22:09.931936 (+174183us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.932001 (+    65us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:09.932073 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:09.935375 (+  3302us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:09.935498 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5531,"prepare.queue_time_us":46424,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":6850,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":167804,"threads_started":1}
I20240426 13:22:09.833451 32330 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.928241 32415 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.857462 32303 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.979964 25070 leader_election.cc:398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:22:09.992805 25070 leader_election.cc:401] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:09.984432 25023 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
W20240426 13:22:09.971302 32341 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.001s	sys 0.008s
I20240426 13:22:09.985349 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:09.950489 32110 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:09.972285 32457 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:09.983408 32242 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:09.947117 32168 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.949873 32218 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:09.950289 31930 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:10.086117 32485 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20240426 13:22:10.086531 31930 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.086788 32461 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20240426 13:22:10.089561 32218 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.092576 32352 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.095s
I20240426 13:22:10.094130 32352 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:10.096547 32213 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.096639 32210 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.097651 32403 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.098209 32315 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.098893 32472 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:10.098776 32315 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.099361 32472 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:10.100262 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.007s	sys 0.003s
W20240426 13:22:10.100610 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.007s	sys 0.003s
I20240426 13:22:10.101013 32315 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:09.972275 32209 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.999723 32330 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:09.998648 32103 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 18. Current term is 19.
I20240426 13:22:10.101367 32303 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:09.978085 32423 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.134876 32472 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.139273 25028 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
W20240426 13:22:10.140563 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 115776us
I20240426 13:22:10.024608 32488 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.083564 32242 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:10.143893 32295 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.096963 32231 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:09.978781 32347 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.984120 32496 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.024685 32110 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.083696 32147 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.026515 32353 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:09.998380 32415 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:10.155735 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 179298us
I20240426 13:22:10.141909 31930 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:10.060899 32457 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.131003 32330 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.098245 32495 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20240426 13:22:10.184417 25025 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:10.177515 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:10.189282 32209 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:10.180955 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 198668us
I20240426 13:22:10.176467 32231 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:10.176476 32510 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:22:10.266919 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 398327us
I20240426 13:22:10.151661 32488 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.187781 31930 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:10.270327 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 272310us
I20240426 13:22:10.248883 32499 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:10.253962 32482 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.181413 32147 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:10.277786 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94332.9us
I20240426 13:22:10.249903 32231 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.301399 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.301923 32520 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.021s
I20240426 13:22:10.302886 32514 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:10.302347 32487 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20240426 13:22:10.303450 32514 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.323333 32514 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.293422 32499 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.323966 32147 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.325784 32499 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.299799 32277 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:10.303316 32423 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.326933 32277 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:22:10.329466 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 2957) took 1485 ms. Trace:
I20240426 13:22:10.333302 24951 rpcz_store.cc:276] 0426 13:22:08.844230 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.849032 (+  4802us) service_pool.cc:225] Handling call
0426 13:22:10.329311 (+1480279us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540038f4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:10.329415 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353752960}
I20240426 13:22:10.336571 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.338878 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.291939 32103 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
W20240426 13:22:10.372303 32507 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20240426 13:22:10.373216 32353 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.373592 32110 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.374379 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.252177 32510 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.375209 25098 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:22:10.376670 32415 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:10.378633 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45781.7us
I20240426 13:22:10.373325 32482 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:10.382382 24951 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20240426 13:22:10.331545 32168 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.391808 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2947) took 2512 ms. Trace:
I20240426 13:22:10.401829 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.401648 24948 rpcz_store.cc:276] 0426 13:22:07.879216 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.289483 (+410267us) service_pool.cc:225] Handling call
0426 13:22:08.289785 (+   302us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:09.002731 (+712946us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.391567 (+1388836us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:10.391751 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1626185472}
I20240426 13:22:10.408422 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.376973 32510 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:10.417397 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.401250 32347 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.419584 32110 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:10.419800 31862 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:10.420235 32110 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.420876 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2914) took 4340 ms. Trace:
I20240426 13:22:10.421308 24953 rpcz_store.cc:276] 0426 13:22:06.080227 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:06.759149 (+678922us) service_pool.cc:225] Handling call
0426 13:22:09.661725 (+2902576us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.703835 (+ 42110us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.420688 (+716853us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54000be850. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:10.420821 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6674399104}
I20240426 13:22:10.422369 32110 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.423207 31862 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:10.424122 31862 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.424787 31930 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:10.425081 32462 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.954s	user 0.011s	sys 0.003s
I20240426 13:22:10.426120 32457 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.428694 32518 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.429015 32488 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.429311 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1064) took 2231 ms. Trace:
I20240426 13:22:10.430173 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3421) took 1348 ms. Trace:
I20240426 13:22:10.430886 32330 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.430572 24952 rpcz_store.cc:276] 0426 13:22:09.082046 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.981433 (+899387us) service_pool.cc:225] Handling call
0426 13:22:09.981707 (+   274us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:09.982001 (+   294us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:09.982025 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:09.982427 (+   402us) log.cc:842] Serialized 26 byte log entry
0426 13:22:10.278637 (+296210us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:10.314323 (+ 35686us) log.cc:842] Serialized 14 byte log entry
0426 13:22:10.327407 (+ 13084us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.327490 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:10.429985 (+102495us) raft_consensus.cc:1671] finished
0426 13:22:10.430004 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:10.430124 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":21114,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":14729,"raft.queue_time_us":81206,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":54321,"spinlock_wait_cycles":31790976,"thread_start_us":285535,"threads_started":1,"wal-append.queue_time_us":405756}
I20240426 13:22:10.434867 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:10.399616 32527 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.429502 32511 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:10.427798 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1364) took 2266 ms. Trace:
I20240426 13:22:10.443522 24956 rpcz_store.cc:276] 0426 13:22:08.161037 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.704688 (+543651us) service_pool.cc:225] Handling call
0426 13:22:08.705043 (+   355us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:08.705408 (+   365us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:08.705434 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:08.734828 (+ 29394us) log.cc:842] Serialized 26 byte log entry
0426 13:22:08.997066 (+262238us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:08.997109 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:08.997175 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:10.424427 (+1427252us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:10.427447 (+  3020us) raft_consensus.cc:1671] finished
0426 13:22:10.427478 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:10.427727 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":784,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":26544,"spinlock_wait_cycles":27995904,"thread_start_us":231132,"threads_started":1,"wal-append.queue_time_us":465165}
I20240426 13:22:10.429898 25615 rpcz_store.cc:276] 0426 13:22:08.198068 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.200711 (+  2643us) service_pool.cc:225] Handling call
0426 13:22:08.799750 (+599039us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.842878 (+ 43128us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.429070 (+1586192us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b540063f450. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:10.429241 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1376884992}
I20240426 13:22:10.438566 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.452392 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.387030 32482 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.435973 32527 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.464872 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.391808 32353 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20240426 13:22:10.465581 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.449061 32285 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:10.450078 32339 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:10.448460 31721 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.474153 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.453132 32511 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:10.475180 24948 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20240426 13:22:10.476531 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 1545) took 1351 ms. Trace:
I20240426 13:22:10.452461 32496 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.479894 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:10.479689 25619 rpcz_store.cc:276] 0426 13:22:09.124679 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.124840 (+   161us) service_pool.cc:225] Handling call
0426 13:22:09.125135 (+   295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.490138 (+365003us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:09.490183 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.490238 (+    55us) raft_consensus.cc:1631] Marking committed up to 16
0426 13:22:09.490299 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.476318 (+986019us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:10.476363 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:10.476477 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838779136}
I20240426 13:22:10.465184 31930 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:10.484851 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.492204 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.492874 24951 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:10.493074 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.493971 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.496137 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.496783 32536 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.497534 32534 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.497331 32536 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.498093 32534 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:10.499452 32536 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.500211 32534 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.501343 32511 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.502445 32353 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:10.502905 32353 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.494984 32231 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.504724 32353 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.505331 32482 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.477802 32242 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.506254 32512 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.506412 24950 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:10.506810 32512 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.507473 32242 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.479243 32339 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.481583 31721 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.509137 32512 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:10.509799 32532 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20240426 13:22:10.509908 32339 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.510146 25618 raft_consensus.cc:1203] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Deduplicated request from leader. Original: 15.10->[17.11-19.12]   Dedup: 17.11->[19.12-19.12]
I20240426 13:22:10.510274 25615 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:10.490123 31930 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.510789 25615 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:10.511651 25615 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.512653 25615 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:10.513754 31721 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.514904 31930 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.510797 25618 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20240426 13:22:10.516784 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3422) took 1434 ms. Trace:
I20240426 13:22:10.429257 32518 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.517223 24954 rpcz_store.cc:276] 0426 13:22:09.082900 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.984740 (+901840us) service_pool.cc:225] Handling call
0426 13:22:09.985001 (+   261us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.239939 (+254938us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:10.239988 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:10.240064 (+    76us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:10.240113 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.516582 (+276469us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54000a7dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:10.516630 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:10.516734 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":585586560}
I20240426 13:22:10.518770 32518 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:10.519189 32469 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.286s	user 0.001s	sys 0.000s
I20240426 13:22:10.521617 25615 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20240426 13:22:10.521992 25615 pending_rounds.cc:76] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 14
I20240426 13:22:10.522197 25615 pending_rounds.cc:97] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20240426 13:22:10.522450 25615 raft_consensus.cc:2866] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:10.523871 32527 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.524192 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.524462 25098 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:22:10.524825 24951 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:10.491333 32496 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.526188 32472 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:10.526266 32496 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.526432 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3406) took 2618 ms. Trace:
I20240426 13:22:10.528406 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.528537 24949 rpcz_store.cc:276] 0426 13:22:07.907925 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.288257 (+380332us) service_pool.cc:225] Handling call
0426 13:22:08.333757 (+ 45500us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:09.001032 (+667275us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.526199 (+1525167us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:10.526377 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532619648}
I20240426 13:22:10.529561 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.529646 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.530072 24949 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 20: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:10.530735 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.530814 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.531101 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:10.532264 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.533941 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.534816 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.535753 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.510360 32532 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.536912 32532 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:10.538054 32535 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:10.538919 32535 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:10.539238 32535 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:10.539170 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.540930 32535 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.541596 32528 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:10.542317 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.542098 32528 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.542882 24949 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:10.476562 32525 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.544906 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.553125 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.545899 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.554066 25024 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 13.
I20240426 13:22:10.478871 32285 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.547601 32496 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.548254 31721 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:22:10.551213 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.551846 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.554934 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.555047 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.573578 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.565783 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.567252 31292 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.575366 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.575299 31292 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.569306 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.574270 25028 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20240426 13:22:10.577039 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.560431 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:10.579340 25028 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:10.579849 25028 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:10.580080 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:10.581036 25023 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:10.579727 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.583724 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.542099 32535 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:10.585306 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.586509 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.585472 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.587077 25030 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:10.588595 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.588999 25097 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:10.565868 32217 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:10.584808 25028 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.590287 31292 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.590874 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.593497 32285 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:10.572649 32342 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.573000 32527 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.598042 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.601095 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:10.601614 25104 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:10.601718 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.602939 25618 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20240426 13:22:10.598234 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.597465 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:10.605401 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.606990 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.588151 32512 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:10.597155 32217 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.613147 32217 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:10.596585 32231 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:10.615252 25616 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20240426 13:22:10.615505 32231 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:10.605443 32342 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.618064 24956 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:22:10.618220 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.618448 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.599792 32528 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:10.620754 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.621172 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.622169 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.608482 32527 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.623389 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.628772 24953 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:10.624275 32538 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20240426 13:22:10.611773 32512 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.624590 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.608991 32534 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:10.611470 32227 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20240426 13:22:10.600824 32526 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.566269 32525 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.626142 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.630177 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.632464 24956 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 13:22:10.643376 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.637109 31292 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:10.635811 32538 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.669224 32526 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.690589 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.677026 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.635525 32340 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20240426 13:22:10.687745 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.693928 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:10.697685 25616 raft_consensus.cc:2396] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20240426 13:22:10.638101 32512 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:10.690328 32538 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:10.716871 25616 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20240426 13:22:10.718968 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:10.722558 25030 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:22:10.723048 25030 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:10.727802 32529 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:10.728304 32421 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.728912 32421 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.730079 32312 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.730580 32312 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:10.730901 32421 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:10.731034 32312 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.723778 25030 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.732477 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.732895 25030 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:22:10.733068 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:10.734431 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:10.735200 32421 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:10.741468 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.744560 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.747177 25588 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:10.755697 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.757380 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.762514 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:10.763588 24956 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:10.765018 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:10.768334 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.751331 32530 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:10.740731 32421 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:10.771600 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.777268 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.771469 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.715684 32277 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:10.777912 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:10.780479 24949 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:10.750975 31721 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.779579 32347 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
W20240426 13:22:10.770432 32467 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.369s	user 0.000s	sys 0.004s
I20240426 13:22:10.782794 25030 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20240426 13:22:10.736254 32488 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20240426 13:22:10.789785 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.771203 32343 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:10.795073 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.780772 32421 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:10.799364 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:10.755014 32408 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20240426 13:22:10.782999 32342 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.802430 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:10.790985 32347 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:22:10.808233 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 201372us
I20240426 13:22:10.770193 32524 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.058s
I20240426 13:22:10.799034 32343 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.822614 32343 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.823403 32525 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.826350 32347 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.820526 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:10.786196 31721 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.177362 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:11.169327 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.002s	sys 0.015s
W20240426 13:22:11.179464 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 348143us
I20240426 13:22:11.183373 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:11.188962 25022 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:11.203016 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.205014 24949 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:10.831805 31292 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 19.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20240426 13:22:11.238617 31902 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:11.302057 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.302488 25030 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:10.808554 32168 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:10.905156 32217 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:10.917173 32312 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:10.906574 32363 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:11.317390 32363 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.317523 32499 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:11.318127 32363 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:11.318972 32526 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.319278 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:10.828142 32508 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
W20240426 13:22:10.826375 32522 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.506s	user 0.001s	sys 0.000s
I20240426 13:22:10.917434 32527 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.350551 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.353175 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.354180 24956 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:11.357209 32295 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:22:11.358078 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:10.728403 32529 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.360059 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.349123 32277 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:10.825436 32532 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.303997 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.361747 32529 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.362314 24953 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:22:11.312805 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.002s	sys 0.019s
I20240426 13:22:11.365221 32525 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.366222 32525 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.368913 25619 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 15: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:11.303797 31902 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:11.304564 32168 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:11.336815 32342 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.338791 32225 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.341369 31721 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.342244 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.333561 24948 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 15.
I20240426 13:22:11.304821 32472 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:11.390327 32565 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.390070 32472 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.391474 32565 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.382587 32168 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:11.363847 32532 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:11.400475 32548 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.109s	user 0.000s	sys 0.002s
I20240426 13:22:11.383571 31902 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:11.419164 32342 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.429548 32342 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.430541 32225 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:11.419608 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69421.7us
I20240426 13:22:11.437974 32225 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.438251 32472 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:11.438827 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.012s	sys 0.013s
I20240426 13:22:11.439108 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.441857 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2954) took 3040 ms. Trace:
I20240426 13:22:11.443460 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:11.449205 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:11.450351 32526 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.443291 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.441797 25030 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
W20240426 13:22:11.474071 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.013s	sys 0.013s
I20240426 13:22:11.490734 32472 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:11.491744 32217 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.493217 24956 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20240426 13:22:11.500257 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 1308) took 1007 ms. Trace:
I20240426 13:22:11.500713 25098 rpcz_store.cc:276] 0426 13:22:10.492711 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.492837 (+   126us) service_pool.cc:225] Handling call
0426 13:22:11.499766 (+1006929us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:11.500044 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70670720}
I20240426 13:22:11.503124 25102 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:11.504254 32526 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.506739 32526 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.507367 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1284) took 3285 ms. Trace:
I20240426 13:22:11.507781 25103 rpcz_store.cc:276] 0426 13:22:08.221586 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:08.221826 (+   240us) service_pool.cc:225] Handling call
0426 13:22:08.506749 (+284923us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:08.599611 (+ 92862us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.506978 (+2907367us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54003b06d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:11.507193 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":654491392}
W20240426 13:22:11.509750 32555 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20240426 13:22:11.485915 31721 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:11.424556 32529 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:11.503033 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3580) took 1519 ms. Trace:
I20240426 13:22:11.466641 24955 rpcz_store.cc:276] 0426 13:22:08.401775 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.695591 (+1293816us) service_pool.cc:225] Handling call
0426 13:22:09.695873 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:09.945914 (+250041us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:09.945951 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:09.945995 (+    44us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:09.946047 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.441346 (+1495299us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:11.441427 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.441673 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":574583680}
I20240426 13:22:11.514500 25027 rpcz_store.cc:276] 0426 13:22:09.983768 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.983912 (+   144us) service_pool.cc:225] Handling call
0426 13:22:09.984272 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.152465 (+168193us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:10.152523 (+    58us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:10.152577 (+    54us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:10.152619 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.502605 (+1349986us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:11.502675 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.502832 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386199168}
I20240426 13:22:11.525369 32565 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.529975 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3581) took 1545 ms. Trace:
W20240426 13:22:11.526772 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 696908us
I20240426 13:22:11.530459 25025 rpcz_store.cc:276] 0426 13:22:09.984099 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.984835 (+   736us) service_pool.cc:225] Handling call
0426 13:22:09.985080 (+   245us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.192638 (+207558us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.529452 (+1336814us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400241654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:11.529784 (+   332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457794432}
W20240426 13:22:11.544900 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101869us
I20240426 13:22:11.546212 32552 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:11.546582 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.547322 32565 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:11.547829 32565 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.548261 32565 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:22:11.545527 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96612.3us
I20240426 13:22:11.552865 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:11.554734 32529 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.548288 32217 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:11.556077 32217 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.556550 32217 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
W20240426 13:22:11.557561 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 723635us
I20240426 13:22:11.558081 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.559540 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.560359 25026 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:11.562808 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.563432 25026 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:11.545693 32525 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.556769 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.570065 32524 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:11.570609 32524 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.570984 32524 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:11.512833 32225 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:11.592689 32486 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.593451 32486 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.599406 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.601873 31721 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240426 13:22:11.616641 32529 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.619427 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.621248 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.622678 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.623143 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.591984 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:11.621490 32216 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:11.640393 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48577.8us
I20240426 13:22:11.585211 32225 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:11.643560 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:11.598984 32486 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.607198 32168 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.647864 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.650524 32286 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:11.650720 32288 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:11.654259 31721 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:11.655416 31721 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.655905 31721 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:11.660364 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.651280 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:11.663412 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.664966 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.667536 32225 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:11.661908 32421 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.856s	user 0.008s	sys 0.012s
I20240426 13:22:11.682740 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:11.683660 32491 log.cc:924] Time spent T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 1.574s	user 0.001s	sys 0.001s
I20240426 13:22:11.685240 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.685608 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.686785 32470 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.682672 32421 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.680763 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.689317 25027 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:11.693331 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.694418 25025 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20240426 13:22:11.694864 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3426) took 1243 ms. Trace:
I20240426 13:22:11.695456 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.696377 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.696506 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.697539 32200 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:11.697973 32076 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:11.698789 32312 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:11.698760 32076 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.700971 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3432) took 1180 ms. Trace:
I20240426 13:22:11.661384 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.672044 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 1196) took 1111 ms. Trace:
I20240426 13:22:11.687631 32470 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:11.667876 32512 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.950s	user 0.011s	sys 0.015s
I20240426 13:22:11.701530 31721 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.695264 24952 rpcz_store.cc:276] 0426 13:22:10.451717 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.451859 (+   142us) service_pool.cc:225] Handling call
0426 13:22:10.452170 (+   311us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.477216 (+ 25046us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:10.477250 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:10.477293 (+    43us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:10.477332 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.694443 (+1217111us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b540030c5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:11.694514 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.694710 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57140992}
I20240426 13:22:11.701341 24954 rpcz_store.cc:276] 0426 13:22:10.521109 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.532259 (+ 11150us) service_pool.cc:225] Handling call
0426 13:22:10.532540 (+   281us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:10.532834 (+   294us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:10.532864 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:10.543610 (+ 10746us) log.cc:842] Serialized 26 byte log entry
0426 13:22:11.364627 (+821017us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:11.365185 (+   558us) log.cc:842] Serialized 14 byte log entry
0426 13:22:11.700527 (+335342us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.700671 (+   144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:11.700750 (+    79us) raft_consensus.cc:1671] finished
0426 13:22:11.700762 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.700915 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":335041,"prepare.queue_time_us":7526,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":56,"raft.queue_time_us":336207,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":227,"thread_start_us":820534,"threads_started":1,"wal-append.queue_time_us":362655}
I20240426 13:22:11.711977 32076 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.712988 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.713511 25098 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:11.713657 32492 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:11.713835 25029 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:11.715777 32168 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.699687 32312 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.698333 32200 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.718395 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.677242 32562 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:11.711565 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.727926 32457 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:11.788803 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.787086 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:11.730290 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 1546) took 2514 ms (client timeout 2335 ms). Trace:
I20240426 13:22:11.731567 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3435) took 1205 ms. Trace:
W20240426 13:22:11.732484 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 111309us
I20240426 13:22:11.733582 32486 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:11.742316 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.743477 31862 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:11.744614 32534 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.094s	user 0.016s	sys 0.013s
I20240426 13:22:11.745409 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:11.746250 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43504.9us
I20240426 13:22:11.747547 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.752794 32357 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.755016 32526 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:11.760394 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.776433 25103 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:11.777105 31721 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.779822 24948 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
W20240426 13:22:11.790238 25622 rpcz_store.cc:269] 0426 13:22:09.216097 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.216291 (+   194us) service_pool.cc:225] Handling call
0426 13:22:11.729932 (+2513641us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:11.730094 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1301212416}
I20240426 13:22:11.768464 25624 rpcz_store.cc:276] 0426 13:22:10.508850 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.538088 (+ 29238us) service_pool.cc:225] Handling call
0426 13:22:10.538360 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.538585 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 18
0426 13:22:10.583636 (+ 45051us) log.cc:842] Serialized 14 byte log entry
0426 13:22:11.516432 (+932796us) log.cc:842] Serialized 14 byte log entry
0426 13:22:11.572531 (+ 56099us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:11.582022 (+  9491us) raft_consensus.cc:1631] Marking committed up to 18
0426 13:22:11.612274 (+ 30252us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.614712 (+  2438us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.619582 (+  4870us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":525250,"prepare.queue_time_us":356158,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":520998,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":474,"thread_start_us":480560,"threads_started":3,"wal-append.queue_time_us":755159}
I20240426 13:22:11.774394 32512 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.799432 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.800729 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.789112 32562 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:11.795948 32526 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:11.802237 25101 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
W20240426 13:22:11.809324 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 120800us
I20240426 13:22:11.808596 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.811802 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.813345 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:11.802449 25104 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:11.815618 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.801012 32534 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:11.818676 25030 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:11.820191 32561 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20240426 13:22:11.821533 32470 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.824561 24952 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
W20240426 13:22:11.824796 32556 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20240426 13:22:11.825515 32512 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.819695 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.828485 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.829174 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:11.829409 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.829818 24947 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:11.830562 32535 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.203s	user 0.022s	sys 0.009s
I20240426 13:22:11.831146 32431 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:11.831596 32431 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:11.832638 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.832845 32431 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.833917 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.834410 32431 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:11.798223 32457 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:11.835422 32538 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.079s	user 0.004s	sys 0.026s
I20240426 13:22:11.835951 32538 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:11.838230 32538 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.839725 25103 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:11.840483 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.840509 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.820410 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.831724 25099 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20240426 13:22:11.829865 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.828508 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.845352 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 3273) took 1285 ms. Trace:
I20240426 13:22:11.845891 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.847155 32357 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:11.847576 32357 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:11.841230 31902 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.850001 31902 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.850829 25025 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:11.852114 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 1195) took 1367 ms. Trace:
I20240426 13:22:11.852509 25619 rpcz_store.cc:276] 0426 13:22:10.484276 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.484394 (+   118us) service_pool.cc:225] Handling call
0426 13:22:11.851877 (+1367483us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:11.852047 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2018571520}
I20240426 13:22:11.845762 25099 rpcz_store.cc:276] 0426 13:22:10.559784 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.560158 (+   374us) service_pool.cc:225] Handling call
0426 13:22:11.845093 (+1284935us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:11.845251 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2882500480}
I20240426 13:22:11.790726 24951 rpcz_store.cc:276] 0426 13:22:10.525838 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.525952 (+   114us) service_pool.cc:225] Handling call
0426 13:22:10.526348 (+   396us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.586793 (+ 60445us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:22:10.760558 (+173765us) log.cc:842] Serialized 14 byte log entry
0426 13:22:11.729774 (+969216us) log.cc:842] Serialized 14 byte log entry
0426 13:22:11.730010 (+   236us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:11.730087 (+    77us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:11.730180 (+    93us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:11.731120 (+   940us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:11.731178 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:11.731385 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":516548,"prepare.queue_time_us":16423,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":2557,"raft.queue_time_us":524774,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":249,"spinlock_wait_cycles":138490752,"thread_start_us":575042,"threads_started":2,"wal-append.queue_time_us":576440}
I20240426 13:22:11.806277 32562 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:11.848634 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.863250 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.854578 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.842742 32225 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:11.847980 32312 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.867352 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.867794 31721 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:11.847960 32357 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:11.867983 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.856787 32421 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:11.865123 32363 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.545s	user 0.014s	sys 0.002s
I20240426 13:22:11.890414 25096 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
I20240426 13:22:11.875911 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.892210 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.893834 25026 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 13 FOLLOWER]: Advancing to term 14
W20240426 13:22:11.895624 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75282.2us
I20240426 13:22:11.900169 25617 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:22:11.902159 32457 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.929260 25098 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
W20240426 13:22:11.904747 32466 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.227s	user 0.002s	sys 0.001s
I20240426 13:22:11.905967 32200 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.906442 31721 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:11.906790 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:11.908371 25024 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:11.909308 32565 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.893091 31902 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:11.912267 24948 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:22:11.868595 32526 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.913329 32201 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:11.915210 32553 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.251s	user 0.001s	sys 0.000s
W20240426 13:22:11.943590 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 141227us
I20240426 13:22:11.943979 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.944566 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.944794 32301 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:11.946095 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.946645 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.906409 24952 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:11.928926 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.949111 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.949784 25022 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 15.
I20240426 13:22:11.950698 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.951503 25022 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:11.931083 32312 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:11.952145 25100 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:11.952358 32312 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.938663 31902 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:11.953095 32565 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.953433 31902 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:11.954757 32312 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.931197 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.929875 32217 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.956126 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.956421 25105 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 23: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:11.957096 32217 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:11.957294 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.931977 32219 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:11.932497 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 92218.5us
I20240426 13:22:11.959303 32219 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.916782 32524 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.960441 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.894747 32363 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.961537 32219 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.961897 32524 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.962393 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.962522 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:11.963022 25622 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:11.964237 31862 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:11.916011 32423 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:11.965165 32423 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:11.966981 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:11.967260 32423 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:11.967314 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.968197 32200 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:11.968564 32200 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:11.968755 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.941263 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.946938 25099 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 19: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:11.969601 32535 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:11.970423 31721 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:11.970803 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:11.971134 32535 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:11.971740 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:11.972276 32474 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:11.968858 32200 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:11.972926 25099 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:11.972800 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.973503 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:11.974203 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.964483 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:11.972792 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.938735 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:11.972365 25022 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:11.938827 25102 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:22:11.972426 31721 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:11.977299 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:11.977959 25102 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:22:11.982077 31721 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:11.982882 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:11.984236 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:11.986188 24947 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:11.990060 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:11.991243 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:11.999532 25097 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 14. Current term is 15. Ops: []
I20240426 13:22:11.989470 32286 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:11.989471 32301 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:12.003509 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.004482 25098 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:12.006747 32169 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.007387 32169 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.009570 32169 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.011272 32565 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.011835 32565 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.013007 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.013887 32565 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.014117 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:12.014513 25104 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:12.015038 25098 raft_consensus.cc:2396] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20240426 13:22:12.016892 32261 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Leader election won for term 15
I20240426 13:22:12.017498 32261 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:12.018486 32261 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.019851 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.021653 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:22:12.022332 25098 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20240426 13:22:12.022684 25098 pending_rounds.cc:76] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 12
I20240426 13:22:12.022922 25098 pending_rounds.cc:97] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20240426 13:22:12.023160 25098 raft_consensus.cc:2866] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:12.023470 25098 pending_rounds.cc:97] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20240426 13:22:12.023701 25098 raft_consensus.cc:2866] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:12.024186 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1083) took 1314 ms. Trace:
I20240426 13:22:12.026309 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.027521 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.028152 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.028384 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.028513 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.028627 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.029134 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.029533 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.029759 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.030211 24955 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:12.030474 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.030795 32225 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:12.031203 32225 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:12.031505 32478 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.001022 32219 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:12.031553 32225 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:12.032631 32219 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:12.034027 25100 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20240426 13:22:12.033735 32219 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.032238 32286 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.039330 32218 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
W20240426 13:22:12.040259 32551 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.353s	user 0.004s	sys 0.000s
I20240426 13:22:12.041127 32218 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Stepping down as leader of term 12
I20240426 13:22:12.041486 32218 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:12.042485 32218 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.043587 32225 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.045953 32225 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.049758 32357 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.051702 32357 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.051437 25026 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 14.
I20240426 13:22:12.061867 25031 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
W20240426 13:22:12.052345 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.767s (SENT) [suppressed 6 similar messages]
I20240426 13:22:12.024597 25620 rpcz_store.cc:276] 0426 13:22:10.686279 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.686447 (+   168us) service_pool.cc:225] Handling call
0426 13:22:10.690872 (+  4425us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:10.691276 (+   404us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:10.691317 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:10.707814 (+ 16497us) log.cc:842] Serialized 26 byte log entry
0426 13:22:10.810337 (+102523us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:10.810411 (+    74us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:10.810496 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:11.914474 (+1103978us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:11.974739 (+ 60265us) raft_consensus.cc:1671] finished
0426 13:22:11.974759 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:12.000266 (+ 25507us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":531,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":10970,"spinlock_wait_cycles":91320576,"thread_start_us":94444,"threads_started":1,"wal-append.queue_time_us":947882}
W20240426 13:22:12.032477 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43815.2us
I20240426 13:22:12.058655 25102 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 13.
I20240426 13:22:12.043834 32218 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Advancing to term 13
I20240426 13:22:12.064947 25070 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.065562 32403 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.065788 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.066136 31721 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:12.066951 32200 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.071189 32498 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:22:12.070983 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:12.071758 31721 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.071938 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.070132 25022 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:12.076715 25098 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 12. Current term is 13. Ops: []
I20240426 13:22:12.079545 25029 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:12.081248 32498 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:12.082350 32498 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:12.082841 32403 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.085675 32403 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.087885 31721 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.090242 25624 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:12.093994 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.099347 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.104243 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.106272 24952 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 24: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:12.106806 31721 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
W20240426 13:22:12.107566 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55233.1us
I20240426 13:22:12.108527 31721 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:12.109054 24955 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 13: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:12.106204 32286 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:12.108261 32457 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:12.122844 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.123607 31721 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.131383 32352 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:12.106312 31902 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.120376 32457 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:12.136394 32457 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:22:12.141080 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69276.3us
I20240426 13:22:12.143369 25027 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:12.128746 32562 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.147770 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.141273 32524 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:12.147189 32363 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.155861 25103 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:12.154479 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:12.157138 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.158543 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.160724 32459 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
W20240426 13:22:12.163933 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3598) took 1592 ms (client timeout 1904 ms). Trace:
W20240426 13:22:12.164420 25023 rpcz_store.cc:269] 0426 13:22:10.568884 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.569030 (+   146us) service_pool.cc:225] Handling call
0426 13:22:12.156503 (+1587473us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:12.161517 (+  5014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9654144}
I20240426 13:22:12.165995 32565 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:12.166465 32565 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:12.166855 32565 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:12.164798 32347 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:22:12.158488 32200 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.171480 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.152592 32524 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.161284 32459 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:12.173137 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.169454 32347 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:12.184823 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.185020 32347 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:12.177983 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.185693 25104 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:12.185695 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.186776 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.185089 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 21. Ops: []
I20240426 13:22:12.188243 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.188923 32527 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:22:12.183306 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.190340 25102 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:12.184028 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:12.190941 32521 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20240426 13:22:12.184399 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.188418 32430 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:12.191886 32562 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.193902 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 1046) took 1589 ms. Trace:
I20240426 13:22:12.194465 25028 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:12.184042 31930 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20240426 13:22:12.194496 25024 rpcz_store.cc:276] 0426 13:22:10.604553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.604687 (+   134us) service_pool.cc:225] Handling call
0426 13:22:12.193668 (+1588981us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400415fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:12.193816 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2994336512}
I20240426 13:22:12.189513 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.202558 32562 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.203244 32526 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:12.203514 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.205085 32496 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:12.205243 32524 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.178211 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.189546 32527 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:12.196400 32403 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:12.211910 32403 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:12.212610 32403 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:12.212888 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.192363 32430 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:12.213665 25617 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:12.214747 32430 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.215664 31902 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.177225 32168 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20240426 13:22:12.217347 32168 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:12.217697 32168 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:12.218531 32168 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.208143 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.211099 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.210412 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:12.222868 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45283.2us
I20240426 13:22:12.226154 25097 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:12.224263 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.233152 25099 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20240426 13:22:12.233224 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:12.234722 32339 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:12.234211 24849 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 9 to 11, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:12.235167 32339 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:12.199476 32347 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.235956 25104 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:12.240391 25098 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
W20240426 13:22:12.240782 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67714.2us
I20240426 13:22:12.219719 32168 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:12.241340 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69962.8us
I20240426 13:22:12.240195 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.242717 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.245465 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.245801 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.249531 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.249719 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.235574 32527 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:12.253826 32527 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:12.254243 32527 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:12.243547 25097 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:12.257187 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.261137 25029 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:12.265105 32194 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20240426 13:22:12.241096 32339 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:12.268790 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:12.269503 24951 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:12.270502 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.274595 24947 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 24: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:12.279732 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:12.280478 25028 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:12.282598 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.288842 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:12.293648 32272 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20240426 13:22:12.294394 32589 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20240426 13:22:12.296427 32493 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.291914 32520 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.297164 32493 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.298199 32520 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.299655 32493 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.297148 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 1043) took 1718 ms. Trace:
I20240426 13:22:12.300509 25028 rpcz_store.cc:276] 0426 13:22:10.578652 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:10.578854 (+   202us) service_pool.cc:225] Handling call
0426 13:22:10.579198 (+   344us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:12.168729 (+1589531us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b58002717b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:22:12.280882 (+112153us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:12.297091 (+ 16209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3442721152}
I20240426 13:22:12.193922 32459 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.303597 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.293315 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.289899 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.303817 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:12.308609 25096 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:12.308931 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.317479 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.340325 32304 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:12.340940 32304 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.343021 32304 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.345785 32533 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:12.346307 32533 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:12.352831 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59609.8us
I20240426 13:22:12.353027 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:12.353652 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.368669 25098 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20240426 13:22:12.372545 32520 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.374640 32347 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.373893 24849 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 11 to 13, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 13 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:12.379099 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.390224 25096 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:12.350272 32533 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:12.393270 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75130.8us
W20240426 13:22:12.401932 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52384.7us
I20240426 13:22:12.404336 32461 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:12.405642 32567 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:12.406029 32567 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:12.410071 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56497.6us
I20240426 13:22:12.411111 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.408535 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.426381 25031 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 15. Current term is 16. Ops: []
I20240426 13:22:12.428948 32465 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.432606 32535 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.433321 32528 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:12.436991 32567 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.438789 32567 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:12.456211 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.448468 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:12.461294 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52839.9us
I20240426 13:22:12.462307 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.374910 32457 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.376262 32565 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.418443 32533 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.445700 32194 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:12.477465 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:12.477636 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.488667 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.474869 32217 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.484809 31862 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:12.465595 32465 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
W20240426 13:22:12.503422 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 125229us
I20240426 13:22:12.494833 32520 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:12.506997 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.509723 25618 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:22:12.381980 24849 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:12.504210 32304 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:12.495051 32339 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.523139 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:12.524083 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 235163us
I20240426 13:22:12.526165 32312 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240426 13:22:12.526912 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 249766us
I20240426 13:22:12.527191 32282 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.527410 32312 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Stepping down as leader of term 14
I20240426 13:22:12.527855 32312 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:12.527652 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:12.517823 32301 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:12.505944 32217 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.537396 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.538359 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.539695 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.543773 24956 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20240426 13:22:12.544560 24954 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20240426 13:22:12.546833 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.548507 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.534071 32168 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:12.554824 24949 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:12.521443 32347 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20240426 13:22:12.559078 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.560719 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.561164 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.561450 24952 raft_consensus.cc:1203] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.10->[17.11-19.12]   Dedup: 17.11->[19.12-19.12]
I20240426 13:22:12.562137 24952 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20240426 13:22:12.558851 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.564424 24947 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:12.540374 32201 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20240426 13:22:12.528195 32465 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20240426 13:22:12.566222 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.543139 32301 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:22:12.570289 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53208.4us
I20240426 13:22:12.575148 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.590348 32169 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.591964 32261 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:22:12.593289 32261 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:12.593869 32261 consensus_queue.cc:1225] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240426 13:22:12.612159 32276 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:12.540182 32312 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.617678 32312 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Advancing to term 15
I20240426 13:22:12.617965 32347 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20240426 13:22:12.618644 32565 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.620225 32495 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20240426 13:22:12.621006 32305 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:12.621497 32305 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:12.621580 32495 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:12.622107 32495 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:12.624588 31721 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:22:12.625015 31721 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:12.622876 32305 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.575474 32225 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:12.626549 32225 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:12.626802 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.627034 32225 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:12.626350 32198 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20240426 13:22:12.631014 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:12.632356 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53353.6us
W20240426 13:22:12.634356 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44618.3us
I20240426 13:22:12.623000 32495 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.634006 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.633253 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:12.559898 32304 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20240426 13:22:12.643508 32478 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240426 13:22:12.645015 32286 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.525s	user 0.012s	sys 0.001s
I20240426 13:22:12.645423 32286 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:12.645961 25030 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:12.646124 31902 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:12.646750 32276 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:12.650315 32528 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:12.568619 32201 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:12.651784 32201 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:12.652544 24949 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:22:12.653013 24949 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:12.653888 24949 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:12.655058 24949 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:12.658165 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3420) took 3871 ms. Trace:
I20240426 13:22:12.658633 24950 rpcz_store.cc:276] 0426 13:22:08.658100 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:09.706427 (+1048327us) service_pool.cc:225] Handling call
0426 13:22:10.329984 (+623557us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:10.506846 (+176862us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:12.340090 (+1833244us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540038f4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:12.515223 (+175133us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b540038f4d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:12.529155 (+ 13932us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1736968832}
I20240426 13:22:12.539537 32403 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.646683 31902 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:12.638796 25615 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:12.625875 31721 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.669988 24956 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:12.670451 24956 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:12.585335 32457 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:12.694461 24956 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.695442 24956 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:12.696653 32305 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:12.673727 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.678308 31721 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:12.680229 31902 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.690351 32403 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.709132 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.669093 32304 consensus_queue.cc:1225] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20240426 13:22:12.692780 32217 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.693789 32339 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.705168 25096 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:22:12.710654 25024 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20240426 13:22:12.713215 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.721793 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.701123 32200 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.015s
I20240426 13:22:12.724632 32565 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:12.725126 32565 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.727480 32565 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.733836 25624 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 24. Current term is 25. Ops: []
I20240426 13:22:12.735265 24954 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20240426 13:22:12.736088 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.737787 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.740476 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.741861 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.745556 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.746361 24950 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:12.748085 32363 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:12.691650 32527 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.749943 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.751036 32528 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:12.751070 32527 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:12.751704 32528 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:12.751967 32276 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:12.719372 32465 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:12.753142 32528 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:12.719617 24849 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 7 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:12.755000 32478 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:12.755427 32478 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:12.756321 32478 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.757589 32478 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:12.768857 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.772059 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:12.772385 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.772622 25096 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:12.773947 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:12.783388 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.784931 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.786130 25029 raft_consensus.cc:1203] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.10->[17.11-20.13]   Dedup: 17.11->[18.12-20.13]
I20240426 13:22:12.786756 25029 pending_rounds.cc:76] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 11
I20240426 13:22:12.787011 25029 pending_rounds.cc:97] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20240426 13:22:12.787310 25029 raft_consensus.cc:2866] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:12.714110 32495 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:12.787986 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.788815 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.789603 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.753530 32528 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:12.793427 24956 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20240426 13:22:12.792929 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:12.835700 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.842115 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:12.836604 25027 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:12.833787 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:12.842893 25028 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:12.847756 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.844372 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:12.849300 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56451us
I20240426 13:22:12.854125 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:12.855485 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:12.858836 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:12.861128 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 139418us
I20240426 13:22:12.862087 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.864427 25030 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
W20240426 13:22:12.866217 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114184us
I20240426 13:22:12.867501 24995 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.870256 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:12.788466 32169 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.835276 32493 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:12.872556 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:12.841398 32168 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:12.784421 32526 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20240426 13:22:12.880801 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:12.889573 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100845us
I20240426 13:22:12.892628 24948 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20240426 13:22:12.890230 31721 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:12.896939 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:12.874114 32339 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:12.920070 32339 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:12.921183 32339 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.925477 32528 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:12.926981 32304 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:12.874285 32521 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.869359 32457 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:12.930642 31702 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:12.931290 31702 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:12.933156 31702 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.934818 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:12.940189 32363 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:12.877607 32357 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:12.897423 32459 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:12.915035 25105 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 16.
I20240426 13:22:12.943306 32567 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20240426 13:22:12.953583 32565 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:12.954136 32565 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:12.954543 32565 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:12.956441 32200 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:12.957571 32330 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20240426 13:22:12.918794 32201 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:12.963057 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54093.2us
I20240426 13:22:12.884816 32493 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:12.968052 24950 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:22:12.972174 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:12.962791 32282 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:12.785759 32403 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.176s
I20240426 13:22:12.993872 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:12.995168 32201 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:12.990999 32493 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.002288 32493 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:13.011827 32528 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:12.984465 24952 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:22:12.971815 32357 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:12.962239 32225 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.012830 31702 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:13.024334 31702 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:12.983232 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 205930us
I20240426 13:22:13.026041 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:12.976114 31721 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:13.028609 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.035931 24947 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20240426 13:22:13.037741 31702 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.058130 31702 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:13.045729 32459 raft_consensus.cc:2741] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20240426 13:22:13.092295 32330 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20240426 13:22:13.110275 32518 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20240426 13:22:13.117717 31862 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.000s
I20240426 13:22:13.118201 31862 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:13.067234 32357 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.095288 32227 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20240426 13:22:13.122288 32330 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Stepping down as leader of term 15
I20240426 13:22:13.148269 25105 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:22:13.156939 31721 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.157574 31721 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.159693 31721 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.162889 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:13.155179 25102 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.010s	sys 0.024s
W20240426 13:22:13.173092 25102 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.011s	sys 0.024s
I20240426 13:22:13.173152 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.152276 32330 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:13.184104 25030 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:13.186806 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:13.169605 32567 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:13.191651 32499 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.028s	sys 0.011s
I20240426 13:22:13.190737 32558 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:13.235983 32194 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:13.242550 32499 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.029s	sys 0.011s
W20240426 13:22:13.247439 24953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.004s	sys 0.031s
I20240426 13:22:13.245524 24950 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 15. Current term is 16. Ops: []
I20240426 13:22:13.251401 24950 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20240426 13:22:13.239915 32169 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:13.251044 24953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.004s	sys 0.031s
I20240426 13:22:13.245606 32558 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.248193 32168 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:13.236621 32567 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:13.268203 32209 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.006s	sys 0.054s
I20240426 13:22:13.238269 32403 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20240426 13:22:13.261296 32169 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.277690 25622 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:13.274561 32558 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:13.277155 32209 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s	user 0.006s	sys 0.054s
I20240426 13:22:13.272763 32225 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:13.247282 32194 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.275244 32217 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
W20240426 13:22:13.290019 32498 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.043s	user 0.017s	sys 0.003s
W20240426 13:22:13.306478 25097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.011s	sys 0.023s
W20240426 13:22:13.307006 25097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.012s	sys 0.024s
W20240426 13:22:13.305446 32498 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.017s	sys 0.003s
I20240426 13:22:13.311425 32527 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:13.278586 32330 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:13.312474 32430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.004s	sys 0.014s
I20240426 13:22:13.330539 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:13.328991 32430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.004s	sys 0.015s
I20240426 13:22:13.346386 31721 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20240426 13:22:13.370532 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.003s	sys 0.023s
W20240426 13:22:13.371039 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.004s	sys 0.023s
I20240426 13:22:13.374058 25103 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:13.377871 25096 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:13.378311 25105 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:13.324733 32169 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:13.344364 31902 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.038s	sys 0.031s
I20240426 13:22:13.379925 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.375036 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:13.395378 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:13.397449 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:13.399430 31721 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:13.400133 31721 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.400534 31721 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:13.362836 32567 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.404294 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:13.363045 32498 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.409267 25101 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:13.398358 31862 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:13.402323 32407 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.015s	sys 0.027s
I20240426 13:22:13.327136 32533 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
W20240426 13:22:13.415827 32598 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20240426 13:22:13.416290 32498 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:13.399554 31902 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.038s	sys 0.032s
I20240426 13:22:13.424727 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:13.448246 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.449039 25027 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:13.449946 31930 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:13.449535 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:13.455742 25028 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:22:13.428349 31862 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:13.426872 32407 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.015s	sys 0.028s
I20240426 13:22:13.446975 32498 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:13.438488 24948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.032s
I20240426 13:22:13.453012 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3300) took 1149 ms. Trace:
I20240426 13:22:13.467399 25103 rpcz_store.cc:276] 0426 13:22:12.303751 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.303873 (+   122us) service_pool.cc:225] Handling call
0426 13:22:12.312921 (+  9048us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:13.374633 (+1061712us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.452775 (+ 78142us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000e9f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:13.452936 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2430573824}
I20240426 13:22:13.469126 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:13.450749 25101 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
I20240426 13:22:13.460927 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:13.473107 24948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.007s	sys 0.032s
I20240426 13:22:13.471478 24950 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:13.475944 32194 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.481436 31721 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.491065 31721 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.505201 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.507397 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:13.431972 32225 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
W20240426 13:22:13.511965 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.030s	sys 0.031s
I20240426 13:22:13.482476 31902 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader election won for term 16
I20240426 13:22:13.506084 25028 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:22:13.502081 25029 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.011s	sys 0.015s
W20240426 13:22:13.514856 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.030s	sys 0.031s
W20240426 13:22:13.520843 25029 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.012s	sys 0.015s
I20240426 13:22:13.529781 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:13.530417 25022 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:13.513437 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.018s	sys 0.009s
W20240426 13:22:13.533268 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.018s	sys 0.009s
I20240426 13:22:13.499101 32496 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:13.533707 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:13.544646 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:13.511808 32305 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20240426 13:22:13.542021 32194 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.547433 32194 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.547922 31930 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:13.549855 32194 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:13.546931 32595 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20240426 13:22:13.553963 32565 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.000s
I20240426 13:22:13.554530 32565 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.560403 32496 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.562891 32496 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.566005 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 2995) took 1032 ms. Trace:
I20240426 13:22:13.567373 32209 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:13.566534 24953 rpcz_store.cc:276] 0426 13:22:12.533554 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.533670 (+   116us) service_pool.cc:225] Handling call
0426 13:22:12.542918 (+  9248us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:12.559087 (+ 16169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:12.559114 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:12.651160 (+ 92046us) log.cc:842] Serialized 26 byte log entry
0426 13:22:13.254886 (+603726us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:13.254923 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.254980 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:13.565726 (+310746us) raft_consensus.cc:1671] finished
0426 13:22:13.565756 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:13.565938 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":603317,"threads_started":1,"wal-append.queue_time_us":912598}
I20240426 13:22:13.579783 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.587507 25027 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:13.595013 32527 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:13.595491 32527 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
W20240426 13:22:13.597469 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.033s	sys 0.037s
W20240426 13:22:13.599330 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.034s	sys 0.037s
I20240426 13:22:13.599167 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:13.604620 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:13.605429 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:13.533854 32543 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.572863 32217 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:13.568095 32209 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:13.626539 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:13.632483 25022 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:13.613281 32194 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:13.656661 31862 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:13.657246 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.027s	sys 0.039s
I20240426 13:22:13.621942 32526 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:13.624150 32217 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:13.660653 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.028s	sys 0.039s
I20240426 13:22:13.661645 32526 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:13.664893 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:13.652160 32567 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20240426 13:22:13.679080 25024 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.020s	sys 0.014s
W20240426 13:22:13.679728 25024 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.021s	sys 0.014s
I20240426 13:22:13.666875 32217 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:13.685879 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 809224us
I20240426 13:22:13.625134 32543 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.677019 32526 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:13.685976 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.001s	sys 0.042s
I20240426 13:22:13.695206 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:13.695693 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.001s	sys 0.042s
I20240426 13:22:13.695854 25030 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:13.679111 32567 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:13.696277 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 828794us
I20240426 13:22:13.691202 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.685282 32565 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.665910 32194 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:13.691928 32493 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.001s
W20240426 13:22:13.689157 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.022s
W20240426 13:22:13.700701 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.018s	sys 0.042s
W20240426 13:22:13.701771 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.019s	sys 0.042s
W20240426 13:22:13.701716 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.013s	sys 0.022s
W20240426 13:22:13.701424 24954 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.011s	sys 0.039s
W20240426 13:22:13.704316 24954 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.011s	sys 0.040s
W20240426 13:22:13.704675 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 536237us
I20240426 13:22:13.707156 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:22:13.702383 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.011s	sys 0.036s
W20240426 13:22:13.712708 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.012s	sys 0.036s
I20240426 13:22:13.693578 32604 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:13.693011 32330 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:13.692152 32563 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
W20240426 13:22:13.712978 32339 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.000s	sys 0.050s
I20240426 13:22:13.697794 32567 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:13.741871 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:13.724910 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:13.725180 32495 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.836s	user 0.010s	sys 0.002s
W20240426 13:22:13.780398 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89299.7us
I20240426 13:22:13.781291 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.724211 32533 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:13.732215 32169 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
W20240426 13:22:13.761667 32339 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.000s	sys 0.051s
I20240426 13:22:13.782639 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.727391 32604 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:13.769876 32227 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20240426 13:22:13.800330 32493 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:13.809173 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.808835 32604 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.811468 32493 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:13.812155 32493 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:13.798243 32168 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:13.813241 32493 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:13.813299 32168 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.815409 32168 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:13.815886 32602 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20240426 13:22:13.816064 32543 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:13.815408 32357 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:13.817338 31702 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:13.817827 32357 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:13.817909 31702 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:13.818930 31702 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.820132 31702 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:13.820715 32194 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:13.824669 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:13.830797 32357 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.834074 32357 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:13.799019 32528 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.763s	user 0.010s	sys 0.002s
I20240426 13:22:13.873746 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3307) took 1101 ms. Trace:
W20240426 13:22:13.885286 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66289.7us
I20240426 13:22:13.889079 32403 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:13.875430 32496 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:13.890339 32496 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.892522 25622 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20240426 13:22:13.892694 32496 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:13.798501 32169 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:13.898509 32567 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.904450 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 3002) took 1350 ms. Trace:
I20240426 13:22:13.881604 25097 rpcz_store.cc:276] 0426 13:22:12.771991 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.772735 (+   744us) service_pool.cc:225] Handling call
0426 13:22:12.773049 (+   314us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:12.773421 (+   372us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:12.773451 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:12.773874 (+   423us) log.cc:842] Serialized 26 byte log entry
0426 13:22:13.307406 (+533532us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:13.307451 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.307581 (+   130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:13.853623 (+546042us) raft_consensus.cc:1671] finished
0426 13:22:13.853652 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:13.873658 (+ 20006us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":158,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":93,"thread_start_us":533173,"threads_started":1,"wal-append.queue_time_us":890880}
I20240426 13:22:13.699568 32209 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:13.886741 32594 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20240426 13:22:13.904870 24955 rpcz_store.cc:276] 0426 13:22:12.554053 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.559726 (+  5673us) service_pool.cc:225] Handling call
0426 13:22:12.578470 (+ 18744us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:12.633581 (+ 55111us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:12.633609 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:12.634093 (+   484us) log.cc:842] Serialized 26 byte log entry
0426 13:22:13.371648 (+737555us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:13.371698 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.371767 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:13.901894 (+530127us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:13.904126 (+  2232us) raft_consensus.cc:1671] finished
0426 13:22:13.904155 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:13.904392 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52596,"raft.queue_time_us":48615,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":16571,"spinlock_wait_cycles":24871168,"thread_start_us":656122,"threads_started":1,"wal-append.queue_time_us":1034108}
I20240426 13:22:13.918872 25030 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:13.801820 32533 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:13.900206 32169 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:22:13.923213 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 378631us
W20240426 13:22:13.795042 32499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.008s	sys 0.008s
W20240426 13:22:13.925874 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 389341us
I20240426 13:22:13.914656 32495 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20240426 13:22:13.935279 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.002s	sys 0.036s
W20240426 13:22:13.935760 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.003s	sys 0.036s
I20240426 13:22:13.913626 32609 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.020s
I20240426 13:22:13.949443 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 1064) took 1180 ms. Trace:
I20240426 13:22:13.943730 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:13.943872 25097 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20240426 13:22:13.947329 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:13.952744 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1398) took 1099 ms. Trace:
I20240426 13:22:13.980957 24949 rpcz_store.cc:276] 0426 13:22:12.853020 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.878412 (+ 25392us) service_pool.cc:225] Handling call
0426 13:22:12.878931 (+   519us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:12.879282 (+   351us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:22:13.380128 (+500846us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.950201 (+570073us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.951262 (+  1061us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.952399 (+  1137us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:13.952446 (+    47us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:13.952497 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.952567 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:13.952688 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":802133,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":17037,"raft.queue_time_us":2108,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":205,"thread_start_us":1049173,"threads_started":2}
I20240426 13:22:13.951485 25031 rpcz_store.cc:276] 0426 13:22:12.769197 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.777770 (+  8573us) service_pool.cc:225] Handling call
0426 13:22:12.843582 (+ 65812us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:12.869690 (+ 26108us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:12.869734 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:12.873455 (+  3721us) log.cc:842] Serialized 26 byte log entry
0426 13:22:13.035454 (+161999us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b180057a2a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:22:13.602687 (+567233us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:13.904791 (+302104us) spinlock_profiling.cc:243] Waited 970 us on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:13.946661 (+ 41870us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.947560 (+   899us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.947647 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:13.947690 (+    43us) raft_consensus.cc:1671] finished
0426 13:22:13.947701 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:13.949390 (+  1689us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38895,"prepare.queue_time_us":320785,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":47281,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":6473,"spinlock_wait_cycles":139467264,"thread_start_us":884170,"threads_started":2,"wal-append.queue_time_us":916213}
I20240426 13:22:13.948984 31902 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:13.972270 32617 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:13.999018 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:13.999819 25618 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.000892 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.001631 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:14.002633 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.003255 24955 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.003271 25624 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:14.007792 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.008472 24947 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:14.008549 24956 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20240426 13:22:14.010043 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:13.972990 32407 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:14.020335 32407 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.021384 32407 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.022704 32407 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.023582 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:14.024901 32613 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20240426 13:22:14.034034 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.034853 25624 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.045822 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.057744 31702 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.058418 31702 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:13.985795 32624 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:14.067493 31702 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.028100 32567 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:14.070971 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 259712us
I20240426 13:22:14.037640 32528 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.073173 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.080129 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.069777 32624 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:14.068459 32565 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.076352 32567 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.087782 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.063745 32217 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.039702 32617 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.121657 31702 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:14.119395 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 324872us
I20240426 13:22:14.122629 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.083854 32357 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:22:14.087821 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 307151us
I20240426 13:22:14.123287 24949 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
W20240426 13:22:14.136150 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48459us
I20240426 13:22:14.086930 32225 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.123718 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.137079 25588 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:14.129132 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.138218 32459 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20240426 13:22:14.138228 32619 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20240426 13:22:14.154589 32459 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.155053 32459 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:14.155551 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3494) took 1031 ms. Trace:
I20240426 13:22:14.078145 32528 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.156044 24952 rpcz_store.cc:276] 0426 13:22:13.124286 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:13.126218 (+  1932us) service_pool.cc:225] Handling call
0426 13:22:13.136298 (+ 10080us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:13.159952 (+ 23654us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:13.160020 (+    68us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:13.160516 (+   496us) log.cc:842] Serialized 26 byte log entry
0426 13:22:13.713852 (+553336us) raft_consensus.cc:1631] Marking committed up to 10
0426 13:22:13.713887 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.713948 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:14.155325 (+441377us) raft_consensus.cc:1671] finished
0426 13:22:14.155347 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:14.155480 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":861,"raft.queue_time_us":12785,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":11790,"thread_start_us":521959,"threads_started":1,"wal-append.queue_time_us":781824}
I20240426 13:22:14.157254 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 1404) took 1193 ms. Trace:
I20240426 13:22:14.136291 32533 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.158573 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 1065) took 1389 ms. Trace:
I20240426 13:22:14.158685 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.157639 24954 rpcz_store.cc:276] 0426 13:22:12.964067 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.965050 (+   983us) service_pool.cc:225] Handling call
0426 13:22:12.973215 (+  8165us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:13.705100 (+731885us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:13.705145 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:13.705689 (+   544us) log.cc:842] Serialized 26 byte log entry
0426 13:22:14.069196 (+363507us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:14.069233 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:14.069295 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:14.155749 (+ 86454us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:14.156963 (+  1214us) raft_consensus.cc:1671] finished
0426 13:22:14.156988 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:14.157190 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2470,"raft.queue_time_us":10858,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"spinlock_wait_cycles":22712704,"thread_start_us":1028342,"threads_started":2,"wal-append.queue_time_us":421513}
I20240426 13:22:14.160466 32604 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:14.158991 25029 rpcz_store.cc:276] 0426 13:22:12.769529 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.778164 (+  8635us) service_pool.cc:225] Handling call
0426 13:22:12.778537 (+   373us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:22:12.893172 (+114635us) raft_consensus.cc:1520] Early marking committed up to index 11
0426 13:22:13.521480 (+628308us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.945239 (+423759us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:13.948656 (+  3417us) log.cc:842] Serialized 52 byte log entry
0426 13:22:13.949926 (+  1270us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:13.950694 (+   768us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.954083 (+  3389us) log.cc:842] Serialized 14 byte log entry
0426 13:22:13.971459 (+ 17376us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:13.971541 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:14.149934 (+178393us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b1800068800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:22:14.158351 (+  8417us) raft_consensus.cc:1671] finished
0426 13:22:14.158372 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:14.158519 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45466,"prepare.queue_time_us":1031815,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":168102,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":20775,"spinlock_wait_cycles":13305088,"thread_start_us":867753,"threads_started":2,"wal-append.queue_time_us":560792}
I20240426 13:22:14.161917 32526 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.163084 25102 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.080852 32305 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20240426 13:22:14.154162 32217 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.089723 32565 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.151574 32617 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.151607 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.086608 32624 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:14.171458 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:14.087718 32563 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.088994 32363 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:14.159602 32498 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20240426 13:22:14.167219 32527 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.000s
I20240426 13:22:14.121312 31862 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Stepping down as leader of term 17
W20240426 13:22:14.167578 32558 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.868s	user 0.008s	sys 0.002s
W20240426 13:22:14.200340 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76685us
I20240426 13:22:14.157724 32528 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.173576 32606 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.173444 32628 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:14.163937 32526 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.117399 32567 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:14.209621 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58076.3us
I20240426 13:22:14.211755 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.202344 32634 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:14.205025 32527 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.214087 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.215157 32527 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.215894 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.202504 32209 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:22:14.202104 32465 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:14.197435 32363 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:14.217918 32465 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.218075 32363 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.218327 32465 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.216493 32610 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:14.202600 31930 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:14.219610 31930 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:14.220469 31930 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.217031 32634 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:14.202234 32498 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.203240 31862 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:14.124517 31902 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.221889 31930 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.174222 32225 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.233382 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.222740 32606 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.235363 32563 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.235388 32637 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.234231 32634 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:14.237380 32606 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.239352 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.232671 32610 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:14.237298 32637 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.241235 32610 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.241670 32610 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.242488 32637 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.243553 32610 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.247614 32565 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.247736 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.249827 32567 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.249912 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.235030 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:14.234902 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 455908us
I20240426 13:22:14.226291 32498 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.261982 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.262944 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.248389 25029 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:14.264876 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.266216 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.266988 25097 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:14.270684 25105 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 17.
I20240426 13:22:14.271090 25098 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:14.272081 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.263396 32533 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.274493 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.275063 25024 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:14.274684 32533 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.276811 32533 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.274502 25101 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: []
I20240426 13:22:14.279050 32624 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.208763 32558 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.281201 32609 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:14.283149 31702 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:14.283954 32533 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240426 13:22:14.237638 32634 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:14.284821 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.285400 31862 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.303400 32634 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:14.287920 32493 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:14.288937 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.267055 32357 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.222385 32628 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.309383 32459 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.310859 32628 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.309379 32357 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.312572 32459 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.291303 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.313359 32357 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.313303 32624 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.314445 32624 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:14.291499 32430 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.291779 32626 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.316242 32430 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.316329 32626 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.318327 32430 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.318913 32626 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:14.300208 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67936.5us
I20240426 13:22:14.319072 32634 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.321022 32634 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:14.325172 32647 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20240426 13:22:14.327020 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.326840 32498 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.327435 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.327646 24956 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.327905 25103 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:14.328060 24948 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:14.297415 25024 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.302340 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:14.333376 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.333621 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.294224 32610 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:14.251852 32169 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.336115 32635 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:14.336208 31902 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:14.336632 32635 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.336767 31902 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.336910 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.337242 31902 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.337572 24954 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:14.338301 32606 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:14.339133 32606 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:14.339432 32606 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:14.331790 32558 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.336997 32169 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.308738 31862 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:14.326946 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.380059 24952 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 24
W20240426 13:22:14.380489 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64757.6us
I20240426 13:22:14.376765 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.357021 24948 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:14.333987 24950 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
W20240426 13:22:14.382402 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 87207.3us
I20240426 13:22:14.343771 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:14.384518 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95665.3us
I20240426 13:22:14.297098 32261 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:14.385568 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.386193 24956 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:14.387042 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.387507 24951 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.388581 24948 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20240426 13:22:14.390151 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.397171 25620 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 25: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:14.376652 32610 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:14.378393 32624 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:14.401212 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 116488us
I20240426 13:22:14.402024 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:14.403167 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.381040 32604 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.404737 24955 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:14.384356 32641 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:14.383613 32635 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:14.408351 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.410384 24952 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.409122 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.389679 32261 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:14.409610 24949 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:14.400453 32474 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.403167 32430 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.418582 32474 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.413323 32459 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20240426 13:22:14.417469 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.418671 32430 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.425956 32430 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.426132 32169 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Leader election won for term 13
I20240426 13:22:14.426884 32169 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.428018 32169 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.400676 32217 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.435838 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.437007 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.406108 32527 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.449730 32527 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.452731 32225 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.453778 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.456157 32217 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.457592 25096 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:14.458519 32217 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.398458 32628 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20240426 13:22:14.460405 32628 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:14.460405 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.460980 32628 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:14.461534 31930 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:14.461917 31930 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.462427 31930 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.461851 32628 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.463431 32628 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.465577 32465 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.467600 32465 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.468516 32495 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:14.389665 32606 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.469218 32495 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:14.470219 32606 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:14.468331 32624 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.475080 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.475950 32495 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:14.478207 32363 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.488541 32634 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:14.489316 32474 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.412719 32498 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:14.488646 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71180.3us
I20240426 13:22:14.489265 32634 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:14.492327 32634 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:14.493103 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:14.437620 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:14.494925 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.495818 25030 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20240426 13:22:14.499045 25023 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:14.500161 32209 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:14.500694 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.501124 25029 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:14.412475 32604 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.502293 25026 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:14.415143 32261 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.503266 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.504657 25025 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 25. Current term is 26. Ops: []
I20240426 13:22:14.506580 32634 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.508534 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.509115 25024 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.509218 32650 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:14.506887 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.509940 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.510244 32357 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:14.510397 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.510685 32357 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.511498 32357 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.503296 32604 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.440836 24947 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:14.513679 32626 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:14.514067 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.514207 32626 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.515101 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.515076 32626 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.515862 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:14.513330 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 104277us
I20240426 13:22:14.483927 24954 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.520390 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.521205 25096 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20240426 13:22:14.521669 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.522114 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.522408 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.522146 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.521109 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.523165 25105 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.523263 25103 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.523528 25097 raft_consensus.cc:2354] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:14.521064 25098 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
I20240426 13:22:14.524219 25622 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:14.524767 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.525470 32527 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.526903 32610 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:14.527509 32363 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.527988 25103 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20240426 13:22:14.528054 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.529621 24951 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:14.532300 25028 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 23: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:14.536455 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:14.521273 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.538349 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.553725 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.554934 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.560302 25104 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:14.561164 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.562335 25101 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20240426 13:22:14.563634 25098 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.578547 25024 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:14.582432 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.583034 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:14.584340 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 170022us
I20240426 13:22:14.585788 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:14.586369 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50002.2us
I20240426 13:22:14.588084 25103 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 18: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.593658 25098 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:14.596129 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.600711 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.582309 32604 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.611356 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.582576 32465 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:14.617201 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.617954 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.614599 32465 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.621033 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.624341 25022 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.603780 32625 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:14.623592 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.585496 32169 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20240426 13:22:14.564299 31930 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.600477 32363 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:22:14.622288 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48957.2us
I20240426 13:22:14.627503 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.631806 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.632014 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.632865 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.634565 25023 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.632853 25103 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 13.
W20240426 13:22:14.637429 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50026.8us
I20240426 13:22:14.639577 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:14.628595 32465 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:14.646152 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.642227 25024 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:14.644804 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:14.645601 32610 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:14.647738 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.643258 32305 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:14.648591 32305 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.650416 32305 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.627635 32625 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:14.653225 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.653692 25027 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:14.646981 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.644904 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:14.654866 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.657001 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.628855 32634 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.659698 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.619396 32604 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.660393 25103 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:14.660720 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.662225 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.663812 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.664363 32610 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:14.665089 32610 consensus_queue.cc:1225] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20240426 13:22:14.665606 32610 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:22:14.665905 32610 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.666723 32610 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.667320 32653 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:14.667810 32653 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.668206 32653 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:14.668257 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.668767 32518 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:14.669238 32634 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:14.653210 31862 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:14.664561 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.671860 25023 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 23: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:14.672065 32217 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:14.672142 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.672619 32217 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.672888 32357 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.667896 32610 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:14.672969 32217 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.688938 32499 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.688418 32261 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.689954 25620 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:14.654788 32474 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:14.692281 32261 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:14.620841 31902 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.692456 32474 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.660089 32567 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:14.693153 32567 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.693403 32474 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.695493 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.696149 25096 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.657914 32625 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:14.701069 25099 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20240426 13:22:14.702688 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:14.702603 32632 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:14.705293 31930 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:14.708802 25096 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
W20240426 13:22:14.713438 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 52768.1us
I20240426 13:22:14.714195 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:14.716070 32654 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20240426 13:22:14.715952 25028 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20240426 13:22:14.718063 25026 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:14.718155 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.729990 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.693485 32567 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.732312 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.750562 25026 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:14.748292 25103 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 14.
I20240426 13:22:14.734583 32604 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:14.735404 32648 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20240426 13:22:14.736373 25030 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:14.737972 31902 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:14.724675 32657 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20240426 13:22:14.740815 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.798449 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.741189 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.637151 32363 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.801003 32493 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:14.801441 32493 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.802485 32363 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.748239 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:14.804425 25096 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:14.785357 32527 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:14.816599 25096 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:14.785213 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.785018 31862 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:14.720428 32563 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:14.818116 32563 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.790884 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:14.785236 32653 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.819042 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.793689 32217 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.802175 32493 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.816793 32527 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.819168 25026 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:14.748378 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.767797 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.819734 24955 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20240426 13:22:14.832980 24955 pending_rounds.cc:76] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 14
I20240426 13:22:14.833237 24955 pending_rounds.cc:97] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20240426 13:22:14.820240 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.833720 24954 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:14.820322 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.829093 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.794747 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.818569 32563 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
W20240426 13:22:14.833380 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 161335us
I20240426 13:22:14.840468 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.841647 25026 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.839999 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:14.830843 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 167070us
I20240426 13:22:14.834731 24947 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.833422 24955 raft_consensus.cc:2866] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:14.845407 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.852413 24952 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 22
W20240426 13:22:14.848383 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100066us
I20240426 13:22:14.853549 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.856670 25024 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:14.864593 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.876518 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.865903 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.866654 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:14.871774 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.878547 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.836320 32209 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:14.878144 24956 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:14.885809 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.886339 24956 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.836767 32650 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20240426 13:22:14.856105 32363 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
W20240426 13:22:14.917264 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50720.3us
I20240426 13:22:14.917598 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.918545 31702 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:14.919018 31702 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.896016 24952 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:14.898450 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.898710 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:14.897349 24947 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:14.902947 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:14.923372 24956 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:14.836681 32261 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20240426 13:22:14.906147 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54483.7us
I20240426 13:22:14.913539 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:14.854442 31862 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:14.855958 31930 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:14.914577 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.876804 32493 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:14.928001 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.900836 32632 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:14.929230 25619 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 14: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:14.884867 32528 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:14.937320 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:14.881565 32194 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:14.879952 32641 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.932160 32407 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:14.930218 32305 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:14.943670 24921 leader_election.cc:398] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:14.940675 32499 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:14.944399 31702 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.945843 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.928273 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:14.912412 32610 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:14.838420 32625 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.943480 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.956812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:14.880717 32357 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20240426 13:22:14.980688 32357 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:14.981046 32357 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.981886 32357 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.983315 32357 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:14.958417 24921 leader_election.cc:401] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:14.984292 32194 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.944516 24954 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 24: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:14.984843 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:14.985217 32194 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.985931 32565 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:14.986421 32565 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:14.986641 32626 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:14.986855 32565 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:14.987246 32626 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:14.901065 32635 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.967402 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:14.994855 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 249468us
I20240426 13:22:14.932830 24952 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59632e528afa44ca98a626355e1e4660"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:14.998962 24952 raft_consensus.cc:595] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.969854 32610 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:14.935410 32169 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
W20240426 13:22:15.018509 32663 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:22:15.024518 32610 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.025600 31862 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.029704 25026 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:14.973292 32465 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.965484 32305 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:14.975248 25615 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:14.975330 32653 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:14.976410 32217 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:14.984476 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66296.8us
I20240426 13:22:14.991736 25029 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:15.001502 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:14.950554 31930 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:15.055804 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:15.012231 32626 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.012914 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:15.016585 32625 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:15.033574 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.044476 25022 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20240426 13:22:14.993716 32459 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader election won for term 14
I20240426 13:22:15.052611 25104 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 21. Ops: []
W20240426 13:22:15.044651 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 166167us
I20240426 13:22:15.050629 25030 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:15.084563 25097 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:15.057456 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:15.002267 32609 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:15.053572 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:15.087417 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 219980us
I20240426 13:22:15.087987 25096 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:15.053226 25098 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 24. Current term is 25. Ops: []
I20240426 13:22:15.067924 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.963439 32407 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:14.962610 32641 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:15.063617 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:15.083029 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49540.8us
I20240426 13:22:15.100889 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:14.953925 32527 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.119108 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:15.119603 25620 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:15.083981 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:15.104158 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.043913 32668 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:15.087006 32217 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:15.128341 24921 leader_election.cc:398] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:15.129235 24921 leader_election.cc:401] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:22:15.129431 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 143716us
W20240426 13:22:15.161305 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44865.8us
I20240426 13:22:15.162405 32407 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.127671 32363 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:15.126019 25030 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:15.130189 25101 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:22:15.167690 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:22:15.185446 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101572us
I20240426 13:22:15.191567 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:15.193892 32639 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:15.194519 32639 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:15.127992 32653 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:22:15.238289 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50411.7us
I20240426 13:22:15.249682 24949 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Advancing to term 16
I20240426 13:22:15.185914 32659 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:15.251892 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.258423 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.258821 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:15.257479 25615 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:15.205597 32499 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:22:15.204331 32625 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:15.262512 24954 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
W20240426 13:22:15.271319 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64950.7us
I20240426 13:22:15.195559 32639 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:15.284370 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98744.9us
I20240426 13:22:15.130232 32668 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:15.252831 32635 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:15.252231 32459 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:15.299090 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:15.299319 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.255617 32650 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20240426 13:22:15.309288 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:15.201046 31930 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.312525 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:15.314821 24995 leader_election.cc:401] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20240426 13:22:15.302711 24949 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20240426 13:22:15.273183 32625 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:15.305663 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:15.279913 32305 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:15.323732 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:15.299652 32670 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20240426 13:22:15.277567 32565 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.277868 32659 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20240426 13:22:15.338702 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:15.287668 32632 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20240426 13:22:15.343973 24921 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
I20240426 13:22:15.345109 24921 leader_election.cc:401] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20240426 13:22:15.313186 32498 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:15.344209 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:15.286561 32518 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20240426 13:22:15.348306 25022 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:15.313390 32465 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.313565 32609 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:15.350514 24955 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:15.350536 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.319020 32625 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:15.347270 32407 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:15.379351 24956 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 16.
I20240426 13:22:15.373050 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:15.384653 24952 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:15.282056 32669 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20240426 13:22:15.344508 32474 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:15.345691 32225 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:15.331708 32653 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20240426 13:22:15.380890 24951 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:15.311805 32217 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20240426 13:22:15.389868 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:15.396649 24953 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:15.301118 32363 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.388073 25022 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 25.
I20240426 13:22:15.360388 32670 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:15.395596 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69284.6us
I20240426 13:22:15.400491 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:15.422719 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 321876us
I20240426 13:22:15.394052 32407 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:15.482121 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68541.5us
I20240426 13:22:15.479974 32658 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20240426 13:22:15.480495 32665 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.283s	user 0.000s	sys 0.001s
I20240426 13:22:15.377295 32641 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:15.485836 24949 raft_consensus.cc:595] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:15.493710 24949 raft_consensus.cc:603] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:15.482642 25098 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:15.487936 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:15.382798 32498 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:15.497614 25022 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:15.500010 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:15.503394 24952 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
W20240426 13:22:15.568974 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 69029.5us
I20240426 13:22:15.397660 32659 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.116s
I20240426 13:22:15.397753 32474 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.597438 31702 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:15.487862 32628 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:15.615319 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 303419us
I20240426 13:22:15.595890 24953 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 25.
I20240426 13:22:15.373574 32459 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:15.611368 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:15.613924 31702 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:15.500973 32626 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:15.511590 32430 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:15.607545 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3693) took 1003 ms. Trace:
W20240426 13:22:15.589445 32677 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
W20240426 13:22:15.590708 32685 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
W20240426 13:22:15.588832 32674 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.199s	user 0.000s	sys 0.000s
I20240426 13:22:15.629973 25022 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:15.601336 32652 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:15.568886 32650 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22.
W20240426 13:22:15.625530 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 316318us
W20240426 13:22:15.631322 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 287446us
W20240426 13:22:15.590183 32682 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20240426 13:22:15.632622 31702 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.636654 31862 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.019s
I20240426 13:22:15.596148 32518 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:15.657699 32518 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:15.658100 32518 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:15.659142 32498 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.628083 32628 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.625138 32671 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.044s
W20240426 13:22:15.615978 32493 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.665s	user 0.008s	sys 0.002s
W20240426 13:22:15.643822 25620 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.524s	user 0.006s	sys 0.001s
W20240426 13:22:15.588258 32684 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20240426 13:22:15.634680 25025 rpcz_store.cc:276] 0426 13:22:14.604614 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:14.610007 (+  5393us) service_pool.cc:225] Handling call
0426 13:22:14.624474 (+ 14467us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:14.624753 (+   279us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:14.624789 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:14.632717 (+  7928us) log.cc:842] Serialized 26 byte log entry
0426 13:22:14.938592 (+305875us) raft_consensus.cc:1631] Marking committed up to 17
0426 13:22:14.938639 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:14.938706 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:15.607266 (+668560us) raft_consensus.cc:1671] finished
0426 13:22:15.607295 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:15.607466 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":279951,"threads_started":1,"wal-append.queue_time_us":463595}
I20240426 13:22:15.589236 32565 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.590246 32407 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:15.624913 32604 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:15.692011 25024 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20240426 13:22:15.738345 25096 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
I20240426 13:22:15.735608 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.649770 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:15.747746 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:15.714660 25030 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:22:15.751017 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 26.
I20240426 13:22:15.752532 24953 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:15.761039 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.761955 24948 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 16.
I20240426 13:22:15.764521 25104 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:15.768209 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:15.769115 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.769868 24947 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:15.770936 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:15.773377 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:15.731922 32563 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.878s	user 0.007s	sys 0.000s
W20240426 13:22:15.723714 32357 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.740s	user 0.009s	sys 0.000s
I20240426 13:22:15.789284 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
W20240426 13:22:15.791324 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 154109us
I20240426 13:22:15.646023 32652 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:15.727542 32474 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.671535 32628 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:15.720628 32533 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:15.670270 32668 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.639981 32626 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:15.811054 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 161379us
I20240426 13:22:15.789546 32528 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:15.819510 32261 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:15.819804 32528 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:15.819973 32261 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:15.820294 32528 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:15.820431 32261 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 16 FOLLOWER]: Advancing to term 17
W20240426 13:22:15.821444 32679 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240426 13:22:15.728418 32658 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20240426 13:22:15.815719 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:15.766613 32604 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:15.853508 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.797442 32659 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20240426 13:22:15.793447 32563 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.862936 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:15.869429 32695 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:15.870132 32695 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:15.841171 32407 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20240426 13:22:15.814301 32533 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:15.879081 32533 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:15.883731 32626 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.850663 32652 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:15.901413 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82551.3us
I20240426 13:22:15.902333 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:15.905907 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:15.906611 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:15.913463 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 124260us
I20240426 13:22:15.915253 25031 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 15.
I20240426 13:22:15.871183 32695 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:15.926736 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63903.3us
I20240426 13:22:15.933632 25027 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:15.938385 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:15.928993 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.940876 24956 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20240426 13:22:15.945374 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:15.948534 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.951005 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:15.929343 32225 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.000s
I20240426 13:22:15.954953 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:15.929824 32430 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:15.955322 32698 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:15.955413 32617 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:15.956779 32261 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:15.958207 32567 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.125s	user 0.010s	sys 0.000s
I20240426 13:22:15.917501 32499 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20240426 13:22:15.961263 32604 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:15.964095 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.007s	sys 0.014s
W20240426 13:22:16.009697 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.007s	sys 0.015s
I20240426 13:22:15.943353 32474 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:15.965615 32169 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:15.938984 32407 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:15.970155 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:16.012759 32169 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:15.975464 32689 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:15.986364 32563 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:15.915184 32528 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:15.990805 24951 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20240426 13:22:16.013974 32689 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:16.005291 32225 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:16.011541 32305 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:16.014531 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:15.993417 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:16.022917 32528 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.023530 32668 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.009668 32698 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.027848 32688 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:16.028236 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.029589 32636 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20240426 13:22:16.033423 32699 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.376s	user 0.004s	sys 0.000s
W20240426 13:22:16.035133 32702 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20240426 13:22:16.040138 25026 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:16.043360 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:16.105038 25029 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:16.031798 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:16.044612 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:16.045327 32474 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:16.046173 32625 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.650s	user 0.007s	sys 0.001s
I20240426 13:22:16.047884 32670 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:16.049216 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:16.050005 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:16.050715 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:16.051769 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:16.011935 32567 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.122315 32689 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.123612 32689 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:16.124264 32567 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:16.124178 32474 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.125881 25621 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:22:16.007102 24949 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:16.060416 24953 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 16: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:16.061069 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:16.086637 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60573.5us
I20240426 13:22:16.062054 32305 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.128697 32609 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:16.129164 32609 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.091949 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:16.086167 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dacba3e2733347ecb1304d307310c944"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:16.129926 32305 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:16.130321 24951 raft_consensus.cc:595] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:16.130714 24951 raft_consensus.cc:603] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:16.130949 32609 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.061806 32628 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:16.131520 32628 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.132158 32625 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.133421 32628 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.136814 25022 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
W20240426 13:22:16.139627 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78662.9us
W20240426 13:22:16.102592 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74547us
I20240426 13:22:16.113735 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 3713) took 1372 ms. Trace:
I20240426 13:22:16.145059 25028 rpcz_store.cc:276] 0426 13:22:14.741651 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:14.798216 (+ 56565us) service_pool.cc:225] Handling call
0426 13:22:16.113440 (+1315224us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:16.113669 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1617606912}
I20240426 13:22:16.119436 25619 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:16.120527 25098 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:16.013630 32169 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.121843 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:16.159089 25025 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 15 FOLLOWER]: Advancing to term 17
I20240426 13:22:16.122437 32658 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:16.059533 32261 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:16.111534 32339 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:16.195974 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.197046 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:16.162179 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:16.199211 24995 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:22:16.199612 24995 leader_election.cc:401] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:16.204854 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:16.205679 24951 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:16.209681 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:16.210458 25028 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:16.014235 32430 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:16.212024 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66797.2us
I20240426 13:22:16.202843 25029 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:22:16.219262 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56722.5us
I20240426 13:22:16.221387 25029 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:16.116673 32659 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20240426 13:22:16.104467 32668 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.228483 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.232534 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:16.236529 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:16.136312 32650 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20240426 13:22:16.160251 32709 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:16.162657 32707 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:16.203038 31862 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:16.275111 24953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:16.275741 24953 raft_consensus.cc:595] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:16.276079 24953 raft_consensus.cc:603] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:16.203100 32528 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.118525 32688 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.204437 32339 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:16.281742 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:16.291646 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:16.294413 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:16.239289 32407 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:16.303711 25619 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:16.304677 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:16.305009 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:16.305776 24954 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
W20240426 13:22:16.312623 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67243.8us
I20240426 13:22:16.203236 32658 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.255339 32709 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:16.239634 32625 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:16.284312 32518 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.000s
I20240426 13:22:16.281268 32636 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:16.297140 32339 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:16.307773 32493 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:16.270489 32567 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20240426 13:22:16.327976 25104 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20240426 13:22:16.341794 25104 pending_rounds.cc:76] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 12
I20240426 13:22:16.327684 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.342067 25104 pending_rounds.cc:97] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20240426 13:22:16.222170 32705 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.109s
I20240426 13:22:16.303210 25028 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:16.275242 32225 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20240426 13:22:16.281594 32670 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20240426 13:22:16.342387 25104 raft_consensus.cc:2866] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:16.345631 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.347996 25098 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:16.320956 32709 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:16.382422 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:16.358111 25028 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:16.355271 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:16.295053 32528 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:16.442050 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 99437.2us
I20240426 13:22:16.340976 32636 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:16.441352 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.447484 25621 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: []
W20240426 13:22:16.448843 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 93745.4us
I20240426 13:22:16.283018 32707 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.453708 24954 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Stepping down as leader of term 13
I20240426 13:22:16.469946 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:16.323838 32518 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.469164 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:16.474678 24954 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:16.343572 32493 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:16.502147 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56667.6us
I20240426 13:22:16.488896 24954 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.514300 24954 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 13 FOLLOWER]: Advancing to term 14
I20240426 13:22:16.520534 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.532845 25104 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20240426 13:22:16.538240 24949 raft_consensus.cc:595] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:16.538791 24949 raft_consensus.cc:603] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:16.540786 25026 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:16.543110 25028 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20240426 13:22:16.519905 32558 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:16.540133 32496 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20240426 13:22:16.551440 24954 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20240426 13:22:16.555377 24954 pending_rounds.cc:76] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 11
I20240426 13:22:16.555658 24954 pending_rounds.cc:97] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20240426 13:22:16.555953 24954 raft_consensus.cc:2866] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:16.525331 32650 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:16.529019 32567 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:16.559406 24954 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:16.561017 24951 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:16.561434 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:16.585835 24956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 23
I20240426 13:22:16.579169 24948 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:16.588989 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:16.589699 24948 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:16.581106 24955 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20240426 13:22:16.527850 32698 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.601368 25025 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:22:16.601119 24956 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:16.519111 32671 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.039s
I20240426 13:22:16.619217 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.623270 32636 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.625397 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.628191 24955 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:16.622418 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:16.557475 32708 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:16.640595 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:16.642293 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 209182us
I20240426 13:22:16.643304 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.602577 32604 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17.
I20240426 13:22:16.605122 32706 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:16.616920 32558 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:16.624949 32565 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:22:16.641676 32689 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.000s
I20240426 13:22:16.617496 32668 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.605150 32669 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.645128 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:16.650246 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 108159us
I20240426 13:22:16.667198 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.667747 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.674799 25025 raft_consensus.cc:2354] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23.
I20240426 13:22:16.672325 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:16.603423 32650 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.686100 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:16.690466 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45438.1us
W20240426 13:22:16.697877 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.012s	sys 0.000s
W20240426 13:22:16.698237 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.013s	sys 0.000s
I20240426 13:22:16.708091 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:16.708885 25098 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:16.643285 32688 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.654053 32518 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.657610 32695 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.714257 25025 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20240426 13:22:16.654747 32706 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:16.688493 32225 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.732076 32567 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:16.732263 32705 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:16.719189 32493 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.649343 32708 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.713217 32668 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Leader election won for term 17
W20240426 13:22:16.766584 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 99448.8us
I20240426 13:22:16.709245 32658 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.725193 32465 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:16.729243 25098 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 25.
I20240426 13:22:16.674022 32565 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.690459 32669 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.736810 32695 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.796851 24921 leader_election.cc:398] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23.
I20240426 13:22:16.788713 32558 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.805979 32565 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:16.755903 32528 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.818171 24921 leader_election.cc:401] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:22:16.811582 32725 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20240426 13:22:16.800258 32719 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20240426 13:22:16.812180 32718 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
W20240426 13:22:16.811694 32722 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20240426 13:22:16.798491 32672 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:16.811762 32710 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.379s	user 0.000s	sys 0.001s
I20240426 13:22:16.798875 32707 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.812259 32714 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:16.812269 32716 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
W20240426 13:22:16.812896 32720 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
W20240426 13:22:16.812279 32724 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20240426 13:22:16.813272 32712 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.353s	user 0.000s	sys 0.001s
I20240426 13:22:16.856137 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.913586 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3729) took 1100 ms. Trace:
I20240426 13:22:16.914052 25030 rpcz_store.cc:276] 0426 13:22:15.812699 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:15.812838 (+   139us) service_pool.cc:225] Handling call
0426 13:22:15.813663 (+   825us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:15.817451 (+  3788us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:15.817485 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:15.840744 (+ 23259us) log.cc:842] Serialized 52 byte log entry
0426 13:22:16.254937 (+414193us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:16.254977 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:16.255051 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:16.913307 (+658256us) raft_consensus.cc:1671] finished
0426 13:22:16.913335 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:16.913521 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3419,"raft.queue_time_us":3941,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":386186,"threads_started":1,"wal-append.queue_time_us":462213}
I20240426 13:22:16.809482 32668 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:16.917104 32668 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.811772 32693 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:16.861667 32729 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.919539 32729 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:16.920743 32739 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20240426 13:22:16.922168 32729 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.770828 32225 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:16.865011 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68319.6us
W20240426 13:22:16.835637 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 102519us
I20240426 13:22:16.834229 32465 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:16.812698 32736 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:16.799028 32737 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.896452 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.886546 25101 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20240426 13:22:16.903009 32721 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
I20240426 13:22:16.903311 32727 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20240426 13:22:16.886394 32704 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.376s	user 0.000s	sys 0.001s
I20240426 13:22:16.903643 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.903703 32728 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.181s
W20240426 13:22:16.904559 32711 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20240426 13:22:16.905022 32738 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 22. Current term is 23.
W20240426 13:22:16.905459 32717 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20240426 13:22:16.905920 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.855496 32493 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:16.887426 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.907130 32708 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.925222 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.931396 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.937011 32727 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:16.927549 32707 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.948361 25031 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 18: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:16.949796 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.950568 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.951350 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.951189 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3727) took 1344 ms. Trace:
I20240426 13:22:16.938643 32736 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:16.941784 32658 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:16.955394 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.887507 32709 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.972648 32709 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:16.958827 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:16.974363 32465 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.976948 32565 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:16.977453 32565 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.960965 32707 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.936978 32737 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.978658 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 3036) took 1315 ms. Trace:
I20240426 13:22:16.979318 32565 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:16.979671 32707 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.979784 32695 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:16.952489 32339 consensus_meta.cc:287] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.000s
I20240426 13:22:16.980649 32339 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.979036 24950 rpcz_store.cc:276] 0426 13:22:15.662888 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:15.663183 (+   295us) service_pool.cc:225] Handling call
0426 13:22:15.725931 (+ 62748us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:15.773942 (+ 48011us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:15.773987 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:15.786912 (+ 12925us) log.cc:842] Serialized 26 byte log entry
0426 13:22:16.115044 (+328132us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:16.115101 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:16.115204 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:16.978433 (+863229us) raft_consensus.cc:1671] finished
0426 13:22:16.978464 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:16.978605 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55726,"raft.queue_time_us":50239,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7243,"thread_start_us":272185,"threads_started":1,"wal-append.queue_time_us":493668}
I20240426 13:22:16.981602 25589 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:16.982803 32339 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:16.968258 32658 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.847815 32706 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.969805 32669 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.910367 32714 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.984680 32658 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.973187 25096 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:16.985283 32723 raft_consensus.cc:979] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:16.985757 32723 raft_consensus.cc:1067] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:16.987972 32736 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:16.988986 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.955160 32650 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.991761 32474 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:16.963999 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.982846 32261 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:16.993134 32261 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:16.993978 32261 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:16.967370 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.985752 32714 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.997918 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.980170 32737 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:16.999984 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.000689 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.002051 25097 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 16. Current term is 17. Ops: []
I20240426 13:22:17.022393 24948 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20240426 13:22:17.003142 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.008981 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:17.015522 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48263.8us
I20240426 13:22:17.027053 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.027323 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.017424 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.030053 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:16.955773 25023 rpcz_store.cc:276] 0426 13:22:15.606553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:15.606705 (+   152us) service_pool.cc:225] Handling call
0426 13:22:15.628606 (+ 21901us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:15.628956 (+   350us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:15.628989 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:16.023326 (+394337us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:22:16.023768 (+   442us) log.cc:842] Serialized 26 byte log entry
0426 13:22:16.329172 (+305404us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:16.329204 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:16.329267 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:16.945911 (+616644us) raft_consensus.cc:1671] finished
0426 13:22:16.945941 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:16.951132 (+  5191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":6869,"spinlock_wait_cycles":10268544,"thread_start_us":305118,"threads_started":1,"wal-append.queue_time_us":593424}
I20240426 13:22:16.951891 32727 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:17.028139 25030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59632e528afa44ca98a626355e1e4660"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:22:17.035924 25030 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:17.035885 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.036666 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.037103 25030 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:17.036816 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:16.954746 32746 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:17.046862 24947 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20240426 13:22:17.046171 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:17.046931 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.041749 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.035073 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.050012 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.051613 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.054065 25028 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 23. Current term is 24. Ops: []
I20240426 13:22:17.055567 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:16.949683 32729 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.077s
I20240426 13:22:17.044324 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.058889 25030 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.014914 31902 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:17.028508 32658 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:17.035351 32610 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:17.041357 32518 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.074472 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.077096 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:17.054622 32672 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.060745 32698 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:17.085402 25030 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:16.956883 32734 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.120s
I20240426 13:22:17.084456 32658 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:17.060648 32746 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:17.047835 32737 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20240426 13:22:17.053237 32695 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20240426 13:22:17.106434 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:17.103106 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:17.100250 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.107481 25621 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:17.074137 31902 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:17.128989 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80895us
I20240426 13:22:17.129848 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:17.130421 24947 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:17.130712 24947 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:17.136438 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.137846 32756 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.138063 25619 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20240426 13:22:17.149876 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.122664 32707 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:17.131498 24947 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.166900 24947 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:17.169622 25029 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: []
I20240426 13:22:17.176357 24947 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:17.182420 25621 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:17.182596 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:17.200655 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
W20240426 13:22:17.202159 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.4:40069 timed out after 1.924s (SENT) [suppressed 2 similar messages]
I20240426 13:22:17.105438 32363 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.052s
I20240426 13:22:17.194732 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:17.196902 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:17.193202 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.198376 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.148926 32675 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:17.211894 24956 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 25.
W20240426 13:22:17.213644 32758 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
W20240426 13:22:17.215484 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 135631us
I20240426 13:22:17.223490 32752 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.220559 25026 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 24. Current term is 25. Ops: []
I20240426 13:22:17.166239 32707 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:17.229702 32713 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.704s	user 0.001s	sys 0.006s
I20240426 13:22:17.241995 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3731) took 1425 ms. Trace:
I20240426 13:22:17.246897 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2838) took 1116 ms. Trace:
I20240426 13:22:17.278740 25024 rpcz_store.cc:276] 0426 13:22:16.131464 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.131582 (+   118us) service_pool.cc:225] Handling call
0426 13:22:16.131885 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:16.135387 (+  3502us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:16.135420 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:16.135458 (+    38us) raft_consensus.cc:1631] Marking committed up to 12
0426 13:22:16.135493 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:17.246694 (+1111201us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:17.246745 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:17.246852 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7640320}
W20240426 13:22:17.249362 32609 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20240426 13:22:17.260110 32652 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:17.277626 25022 rpcz_store.cc:276] 0426 13:22:15.816772 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:15.816910 (+   138us) service_pool.cc:225] Handling call
0426 13:22:15.817191 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:16.137276 (+320085us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:17.241746 (+1104470us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54003ae8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:17.241934 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":728169856}
I20240426 13:22:17.292691 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:17.300869 32761 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
W20240426 13:22:16.997449 32726 consensus_queue.cc:1159] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 19 status { last_received { term: 19 index: 11 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 18. Current term is 19. Ops: [18.11-18.11]" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:22:17.318038 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.319504 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:17.328997 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105669us
I20240426 13:22:17.219729 32754 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.290539 32658 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.270047 32669 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:17.372785 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78974.5us
I20240426 13:22:17.375643 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1114) took 1346 ms. Trace:
I20240426 13:22:17.376197 25620 rpcz_store.cc:276] 0426 13:22:16.028920 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.060160 (+ 31240us) service_pool.cc:225] Handling call
0426 13:22:16.103902 (+ 43742us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:16.318314 (+214412us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:16.318343 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:16.318384 (+    41us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:22:16.318412 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:17.375366 (+1056954us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:17.375432 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:17.375578 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492648320}
I20240426 13:22:17.323381 32653 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.319810 32723 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:17.374208 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:22:17.224200 32558 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.136s
I20240426 13:22:17.225143 32752 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.425581 25098 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20240426 13:22:17.444200 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 104522us
I20240426 13:22:17.380503 32499 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:17.448597 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 246440us
I20240426 13:22:17.450892 32746 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:17.425153 31862 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.361613 32754 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.425375 32610 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.067s
I20240426 13:22:17.382484 32707 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.454259   305 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:22:17.455704 32738 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:17.457259 31902 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:17.381292 32658 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.457926 32743 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.458138 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 1112) took 1441 ms. Trace:
I20240426 13:22:17.458349 32715 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:17.458814 32624 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:17.425199 32650 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.459959 32752 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.502661 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.500651 31902 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20240426 13:22:17.503566 32694 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:17.476666 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.511713 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.489426 25622 rpcz_store.cc:276] 0426 13:22:16.017075 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.017208 (+   133us) service_pool.cc:225] Handling call
0426 13:22:16.017477 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:16.134274 (+116797us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:16.134304 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:16.134341 (+    37us) raft_consensus.cc:1631] Marking committed up to 18
0426 13:22:16.134374 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:17.457846 (+1323472us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:17.457910 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:17.458063 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":268228864}
I20240426 13:22:17.382807 32669 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:17.481792 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276641us
I20240426 13:22:17.498582 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.500919 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.515405 25098 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:17.502060 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.477382 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:17.492930 32715 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:17.518558 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.529047 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.474171 31862 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.520429 32741 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.516065 32629 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:17.569125 32743 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:17.570919 32672 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:17.571518 32672 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.572191 32658 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.574165 32672 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.574975 32629 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:17.575492 32629 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:17.510993 32698 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.576345 32225 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.576584 32698 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.577423 32629 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:17.577968 25098 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:17.578584 32698 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:17.580715 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.581461 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.583065 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.523350 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:17.524713 32617 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:17.504534   305 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:17.517132 32625 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.506120 32723 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.597952 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.598129 32723 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:17.538538 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47598.5us
W20240426 13:22:17.540663 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 212737us
I20240426 13:22:17.537671 32752 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:17.537925 32746 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.583221 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.465780 32499 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.509739 32650 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:17.594707 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71460.2us
I20240426 13:22:17.596052 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:17.614307 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 99093.8us
I20240426 13:22:17.596230 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.509739 32694 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.596524 32617 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.593353   305 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:17.571005   310 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20240426 13:22:17.604674 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.607990 25026 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
W20240426 13:22:17.672398 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52600.3us
I20240426 13:22:17.616362 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.585932 32695 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:17.619817 24950 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20240426 13:22:17.622866 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:17.679586 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63323.7us
I20240426 13:22:17.680693 25026 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20240426 13:22:17.599220 31862 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:17.669592 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50031.6us
I20240426 13:22:17.608290 32629 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:17.605406 32752 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:17.675015 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.675444 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.676903 24953 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:17.602439 32698 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20240426 13:22:17.609403 32653 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:17.677595 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.685276 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:17.685940   319 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.690054   314 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:17.692086 32639 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.626610 32407 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20240426 13:22:17.713992 32723 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:17.689220 32650 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.716166 32639 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.717420 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.718231 32639 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:17.718621 32357 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:17.719055 32357 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:17.719415 32357 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:17.702082 32752 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
W20240426 13:22:17.695581   311 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20240426 13:22:17.696105 32626 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:17.590634 32741 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.696370 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3569) took 1115 ms. Trace:
W20240426 13:22:17.696759 32745 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:22:17.697446 32499 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.687287 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:17.706857 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.709863 25031 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20240426 13:22:17.709889   314 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.720510 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.721678 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.733974 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.734588 24948 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:17.722069 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.614413 32746 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.725332 25030 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20240426 13:22:17.725808 24951 rpcz_store.cc:276] 0426 13:22:16.581098 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.592391 (+ 11293us) service_pool.cc:225] Handling call
0426 13:22:16.592781 (+   390us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:16.728506 (+135725us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:16.728536 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:16.768805 (+ 40269us) log.cc:842] Serialized 52 byte log entry
0426 13:22:16.947353 (+178548us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:16.947392 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:17.695953 (+748561us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400161f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:17.696007 (+    54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:17.696072 (+    65us) raft_consensus.cc:1671] finished
0426 13:22:17.696086 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:17.696319 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2781,"raft.queue_time_us":178269,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"spinlock_wait_cycles":304790272,"thread_start_us":102592,"threads_started":1,"wal-append.queue_time_us":170215}
I20240426 13:22:17.721963 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:17.727941 25098 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20240426 13:22:17.728257 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.744230 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.710433 32629 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:17.745117 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.744520 24947 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:17.626230 32715 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.746140   313 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240426 13:22:17.720430 32653 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.751170 32363 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:17.732249   305 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.755741 32697 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:17.722853 32709 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20240426 13:22:17.761162 32617 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.761626 32653 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.762492 32746 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.709815   319 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.764215 32741 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:17.765118   319 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.766335 25624 raft_consensus.cc:2396] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20240426 13:22:17.766934 32681 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.767400 32681 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.769057 32754 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.769327 32740 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:22:17.770076 32697 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:17.773356   314 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.687891 32695 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.778205 32169 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
W20240426 13:22:17.780970 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71382.5us
W20240426 13:22:17.784900   315 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20240426 13:22:17.785128 32695 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.789320 32629 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:17.797955 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61438.6us
I20240426 13:22:17.798745 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:17.799779 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.800416 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.807029 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:17.868141 32669 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.808228 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.900571 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:17.900528 32694 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:17.807595 32499 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.064s
I20240426 13:22:17.811319 32652 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:17.902741 25030 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20240426 13:22:17.900166   329 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.095s	user 0.002s	sys 0.006s
W20240426 13:22:17.909130 32727 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.837s	user 0.005s	sys 0.006s
I20240426 13:22:17.912536 32727 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:17.913400 32209 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:17.913549 25027 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:17.916308 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.916386 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:17.916941 25030 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:17.917027 25104 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 17.
I20240426 13:22:17.904537 32628 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:17.923331 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:17.906860 32459 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:17.910223 32652 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:17.921343   326 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.105s	user 0.006s	sys 0.000s
I20240426 13:22:17.944660 32672 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:17.913610 32681 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:17.948386 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.954356 32738 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:17.951290 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.964428 32738 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:17.965492 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:17.966878 24951 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20240426 13:22:17.967979 24951 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:22:17.968935 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:22:17.955219 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54793.5us
I20240426 13:22:17.930215 32459 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:17.981166 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20240426 13:22:17.953183 32759 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.727s	user 0.006s	sys 0.002s
I20240426 13:22:17.997303 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:17.947995 32652 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:18.003196 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:17.951706 32672 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:18.004972 32672 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:18.013432 32430 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20240426 13:22:18.013798 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 1115) took 1984 ms (client timeout 1924 ms). Trace:
I20240426 13:22:18.014504 32676 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:18.015075 32705 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:18.015273 32639 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:18.016443 32527 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:18.018307 32740 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:18.019050 32636 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:18.029289 32465 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20240426 13:22:18.036774   325 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20240426 13:22:18.038882 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.039549 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.039644 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.040483 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.040550 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.045150 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.052248 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:18.008217 32731 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20240426 13:22:18.056969 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 3762) took 1205 ms. Trace:
W20240426 13:22:18.056360 25624 rpcz_store.cc:269] 0426 13:22:16.029241 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.031544 (+  2303us) service_pool.cc:225] Handling call
0426 13:22:18.013571 (+1982027us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:18.013716 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3812242688}
I20240426 13:22:18.062260 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:18.098994 25028 rpcz_store.cc:276] 0426 13:22:16.851687 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.937182 (+ 85495us) service_pool.cc:225] Handling call
0426 13:22:16.937467 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:17.069343 (+131876us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.056750 (+987407us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400055fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:18.056912 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267583744}
I20240426 13:22:18.069221 24949 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:18.069563 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:18.100943 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48819.3us
I20240426 13:22:18.101670 25096 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 25: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:17.994995 32459 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:18.063026 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:18.101753 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:18.102523 25619 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:18.103957 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:18.071606 32757 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20240426 13:22:18.087898 32689 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.104362 32688 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20240426 13:22:18.073467 32639 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:18.123478 32639 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.124651 32639 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:18.126708 24949 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:22:18.110602   302 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:18.114320 32563 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:18.129014   302 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:18.107761 32567 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20240426 13:22:18.137709 32563 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:18.130517 32689 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.146245   302 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
W20240426 13:22:18.157263 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55604.8us
I20240426 13:22:18.133847 32688 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20240426 13:22:18.146034 32727 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.138774 32672 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.152093 32563 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:18.141582 32357 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.196344 24953 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 25: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:22:18.197052   334 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20240426 13:22:18.198026 32706 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:18.206658   302 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.207881 25105 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 20. Current term is 22. Ops: []
I20240426 13:22:18.209707   302 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:18.219326 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 2851) took 1191 ms. Trace:
I20240426 13:22:18.219734 25029 rpcz_store.cc:276] 0426 13:22:17.027841 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.028067 (+   226us) service_pool.cc:225] Handling call
0426 13:22:17.169391 (+141324us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:17.170171 (+   780us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.219146 (+1048975us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54004164d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:18.219262 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":324208512}
I20240426 13:22:18.221047 24955 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20240426 13:22:18.221529 24955 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 10
I20240426 13:22:18.221817 24955 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20240426 13:22:18.222105 24955 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:18.223703 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3576) took 1224 ms. Trace:
I20240426 13:22:18.233713 24955 rpcz_store.cc:276] 0426 13:22:16.998794 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.026703 (+ 27909us) service_pool.cc:225] Handling call
0426 13:22:17.026982 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:18.222610 (+1195628us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.223441 (+   831us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:18.223636 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2745222912}
I20240426 13:22:18.150508 32752 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.225741 32672 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.238704 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.228363 32726 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:18.234771 32704 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20240426 13:22:18.240401 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:18.251760 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.250244 24955 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20240426 13:22:18.258656 32752 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.244797 32689 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.241050 32624 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20240426 13:22:18.264153 32357 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.260048 32706 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:18.267048 32707 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:18.280553 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.267156 32527 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:18.313926 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.315364 25099 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:18.327965 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:18.242961 32726 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.349273 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 3054) took 1271 ms. Trace:
I20240426 13:22:18.349717 24956 rpcz_store.cc:276] 0426 13:22:17.077996 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.214949 (+136953us) service_pool.cc:225] Handling call
0426 13:22:17.215237 (+   288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:17.460632 (+245395us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:17.460663 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:17.461110 (+   447us) log.cc:842] Serialized 26 byte log entry
0426 13:22:17.514511 (+ 53401us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:17.514573 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.334114 (+819541us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540004c754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:18.334168 (+    54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:18.334236 (+    68us) raft_consensus.cc:1671] finished
0426 13:22:18.334252 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:18.349210 (+ 14958us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2539,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":15857,"spinlock_wait_cycles":563540736,"thread_start_us":52167,"threads_started":1,"wal-append.queue_time_us":124239}
I20240426 13:22:18.299881 32465 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20240426 13:22:18.271750 32459 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.340684   305 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:18.278271 32706 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.388360 25620 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:18.394142 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 19 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:18.395967 24852 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 19 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:18.397732 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 20 to 24, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 24 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:18.384903   313 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:18.401553 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:18.402493 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:18.403442 32459 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:18.405515 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 434628us
I20240426 13:22:18.406208 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:18.407773 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 344596us
I20240426 13:22:18.409137   331 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:18.409564   331 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.409901   331 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:18.411521 31930 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.412250 25027 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20240426 13:22:18.412143 31930 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.413446 32726 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.414160 31930 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.414997 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:18.415592 25028 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:18.418960 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:18.419045 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.419679 24955 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:18.422902 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:18.424333 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:18.429309 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:18.429893 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.401640 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:18.431864 25028 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:18.440850 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:18.399320 24852 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 17 to 28. New cstate: current_term: 28 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:18.448305 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.447196 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.449888 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.450675 25025 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:18.451323 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.454144 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.440554 32495 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240426 13:22:18.458140 32625 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:18.458571 32625 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:18.458927 32625 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:18.461876 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.426656 32528 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.475075 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.399202 32563 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.446038 32697 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:18.537173 25099 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:18.544915 24955 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:18.496435 32686 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.550426 32686 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.550818 32652 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.552294 25619 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:22:18.552436 32686 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.551277 32715 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:18.501436 32659 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.556126 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.563596 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:18.566248 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:18.547103 32697 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:18.570605 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:18.484752 32528 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.509759 32639 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20240426 13:22:18.537238 32675 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20240426 13:22:18.605142 25103 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20240426 13:22:18.605692 25103 pending_rounds.cc:76] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 12
I20240426 13:22:18.605969 25103 pending_rounds.cc:97] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20240426 13:22:18.606257 25103 raft_consensus.cc:2866] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:18.580255 32714 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:18.564838 32659 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.591920 32339 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:18.619882 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:18.620851 24949 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:18.621584 32697 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.624747 32339 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:18.625617   347 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20240426 13:22:18.598273 32686 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.626969 32686 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:18.627945   348 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:22:18.628957 32686 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.630439 24949 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 25.
I20240426 13:22:18.636150 31702 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:18.637120 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.653393 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:18.763468 31702 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.781901 31702 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.784914 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:18.660006 32714 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:18.798198 31702 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:18.744304   338 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20240426 13:22:18.741711 32747 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.659307 32652 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:18.742158 32658 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:18.744908 32669 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.742218 32493 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:18.741168 32706 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.818414 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.791271 32563 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.744498 32225 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:18.746515   329 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:22:18.650417   331 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.773243 32617 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:18.834780 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:18.798990 32736 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:18.821094 32747 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.849567 25620 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:18.794973 32625 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.862217 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.864988 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:18.833137 32493 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.867640 32493 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.856639 32617 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.869608 32738 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.870705 32617 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.874192 32625 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:18.877955   349 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20240426 13:22:18.879038   356 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20240426 13:22:18.879763 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.830077 32669 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.842116 32689 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.881656 32706 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:18.881991 32689 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.882488 32706 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:18.881671   355 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:22:18.885759 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1379) took 1103 ms. Trace:
I20240426 13:22:18.887523 32625 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.887635 32734 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.878758 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:18.888532   357 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:22:18.888036 32625 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.888370 32734 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.888898 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.890195 32625 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.880694 25102 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:18.891554 32528 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.828915 32714 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:18.892593 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:18.893524 32735 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:18.893923 32735 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:18.894223 32735 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:18.894768   331 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.890367 32734 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.824976 32658 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.895895 32669 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.899812 32706 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.900287 32658 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.849506 32225 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.907552 32225 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.916160 32659 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:18.920605 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:18.921036 25620 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:18.922610 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:18.923197 25029 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:18.923504 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.926035 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:18.886274 25097 rpcz_store.cc:276] 0426 13:22:17.782526 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.782754 (+   228us) service_pool.cc:225] Handling call
0426 13:22:17.792916 (+ 10162us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:17.802909 (+  9993us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:17.802939 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:17.813869 (+ 10930us) log.cc:842] Serialized 26 byte log entry
0426 13:22:18.426553 (+612684us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b1800476c80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:22:18.582643 (+156090us) raft_consensus.cc:1631] Marking committed up to 11
0426 13:22:18.582689 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.582763 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:18.885400 (+302637us) raft_consensus.cc:1671] finished
0426 13:22:18.885428 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:18.885687 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69030,"raft.queue_time_us":13505,"raft.run_cpu_time_us":7945,"raft.run_wall_time_us":94856,"spinlock_wait_cycles":6448512,"thread_start_us":155875,"threads_started":1,"wal-append.queue_time_us":714363}
I20240426 13:22:18.932441 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:18.934453 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:18.939698 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.941102 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.939805 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.942313 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:18.943650 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:18.946849 25097 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 25: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:18.949891 32493 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:18.950978 32766 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:18.951393 32766 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:18.952232 32766 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.952641 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.953438 32766 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:18.925791 32689 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:18.960165 32709 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:18.953908 32675 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:18.963712 32709 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.964798 32700 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:18.959401 32751 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:18.965948 32751 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:18.966516 32709 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:18.968362 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.970245 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.970819 25102 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 19.
I20240426 13:22:18.971984 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:18.973017 32689 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:18.974109 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:18.974951 32689 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20240426 13:22:18.975319 32689 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:18.975481 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 3373) took 1301 ms. Trace:
I20240426 13:22:18.976150 25103 rpcz_store.cc:276] 0426 13:22:17.673854 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.674034 (+   180us) service_pool.cc:225] Handling call
0426 13:22:17.686322 (+ 12288us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:18.606770 (+920448us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:18.975208 (+368438us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54000ea454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:18.975413 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2112015616}
I20240426 13:22:18.976887 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:18.981209 32700 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:18.982189 32700 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:18.975661 32689 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:18.985385 25029 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 25.
W20240426 13:22:18.993551 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45563.2us
I20240426 13:22:19.000885 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.001781 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.002758 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.002543 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.014617 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.015223 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.015592 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.016255 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:19.014637 25026 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:19.018519 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 20 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:18.858004 32736 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.024238 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.027061 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:19.021636 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.028177 25025 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:19.030990 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.033792 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.027168 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.036358 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:18.994112 32628 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:19.002542 32499 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.035825 25022 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:22:19.012005 32756 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
I20240426 13:22:18.892663 32714 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.029500 32493 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:19.050595 32493 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.052271 32499 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.054234 25024 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:19.054381 32499 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:19.056716 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.057425 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.045560 25026 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:19.057987 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.063048 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.041626 32339 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:19.067108 32735 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:19.071168 24952 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 3050) took 2780 ms (client timeout 2151 ms). Trace:
I20240426 13:22:19.073609 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.074743 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.075752 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.029556 32751 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:19.079149 32357 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:19.079327 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 652767us
I20240426 13:22:19.097311 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:19.098291 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:19.079671   364 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:22:19.081159 32726 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:19.101939 25618 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20240426 13:22:19.102145 32604 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:19.102749 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:19.103200 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:19.082875 32736 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:19.071584 24952 rpcz_store.cc:269] 0426 13:22:16.291198 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:16.291360 (+   162us) service_pool.cc:225] Handling call
0426 13:22:19.070983 (+2779623us) spinlock_profiling.cc:243] Waited 2.57 s on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:19.071087 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5914389888}
I20240426 13:22:19.090641 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.102625 32604 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.107475 32604 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.089890 32628 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.109418 32628 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:19.109546   361 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s	user 0.002s	sys 0.000s
I20240426 13:22:19.110028 25022 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:19.110865 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 3069) took 1426 ms. Trace:
I20240426 13:22:19.111320 24953 rpcz_store.cc:276] 0426 13:22:17.684734 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.706546 (+ 21812us) service_pool.cc:225] Handling call
0426 13:22:19.110582 (+1404036us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:19.110756 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1067780480}
I20240426 13:22:19.112098 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:19.112762 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:19.114284 32747 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.115262 24950 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 24. Current term is 25. Ops: [24.12-24.12]
I20240426 13:22:19.116358 25026 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:19.086812 32493 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:19.116957 25031 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 24. Current term is 25. Ops: [22.11-24.12]
I20240426 13:22:19.120563 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.117960 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.122537 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:19.123169 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.123824 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.124235 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.123217 24948 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:19.125890 24953 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25. Ops: []
I20240426 13:22:19.127458 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.127561 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:19.128163 25099 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:19.129419 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.131367 25097 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 19: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:19.133759 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.133993 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.135879 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.137310 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.137997 25620 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:19.119784 32726 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:19.137504 24954 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20240426 13:22:19.143437 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.143970 25026 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 22
I20240426 13:22:19.143849 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:19.154547 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.153565 32627 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.005s
I20240426 13:22:19.128612 32659 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.134559 32493 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20240426 13:22:19.140628 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 109029us
I20240426 13:22:19.136909 32747 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:19.140446 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:19.170043 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.167297 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.174533 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.167838 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3587) took 1231 ms. Trace:
I20240426 13:22:19.175485 24950 rpcz_store.cc:276] 0426 13:22:17.936612 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.948920 (+ 12308us) service_pool.cc:225] Handling call
0426 13:22:19.114967 (+1166047us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:19.115704 (+   737us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:19.167661 (+ 51957us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400366fd0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:22:19.167785 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586278272}
I20240426 13:22:19.151822 25026 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:19.170764 32610 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:19.165843 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.180370 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.183517 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.183815 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.193311 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.197238 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.203069 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.192698 32357 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:19.167829 32659 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.211557 32659 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.204766 32686 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:19.212663 32707 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:19.213065 32707 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.213439 32707 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:19.182107 32493 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20240426 13:22:19.206293 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.221514 32493 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:19.221172 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.181787 32747 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:19.185163 32225 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:19.226248 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.232628 25023 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:19.207851 32610 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:19.236124 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.236351 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.241778 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.225430 32686 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:19.243891 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:19.244303 32493 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:19.222441 32735 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.253695 25097 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:19.234021 32747 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:19.235299 32225 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:19.259148 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.232190 32740 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20240426 13:22:19.226806 32676 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.035s
I20240426 13:22:19.245625 32610 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:19.248415 32659 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:19.254899 32357 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:19.256835 32641 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:19.299532 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.298591 32659 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.300979 32357 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:19.289896 32225 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:19.297140 32610 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:19.291280 32747 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:19.308689 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.298316 32694 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:19.297566 32735 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:19.313556 25029 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:19.307401 32659 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:19.326730 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.304208 32339 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20240426 13:22:19.330579 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.330960 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.331581 25027 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:19.313091 32694 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:19.334156 24852 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 19, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.340484 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 1377) took 1575 ms. Trace:
I20240426 13:22:19.341692 25097 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:19.343237 25098 rpcz_store.cc:276] 0426 13:22:17.765128 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:17.798727 (+ 33599us) service_pool.cc:225] Handling call
0426 13:22:17.799093 (+   366us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:22:17.799485 (+   392us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:17.799540 (+    55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:22:17.806555 (+  7015us) log.cc:842] Serialized 26 byte log entry
0426 13:22:18.817126 (+1010571us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b18004769e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:22:19.036139 (+219013us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:22:19.036174 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:19.036234 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:19.340115 (+303881us) raft_consensus.cc:1671] finished
0426 13:22:19.340142 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:19.340419 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7912,"raft.queue_time_us":3005,"raft.run_cpu_time_us":7717,"raft.run_wall_time_us":97629,"spinlock_wait_cycles":4014464,"thread_start_us":218832,"threads_started":1,"wal-append.queue_time_us":1465328}
I20240426 13:22:19.261451 32730 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.067s
I20240426 13:22:19.345950 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.351989 25099 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:19.359907 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:19.360755 25102 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:19.361840 32694 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.380528 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:19.427258 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46840.8us
I20240426 13:22:19.447561 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.451061 31862 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:19.427031   368 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20240426 13:22:19.362135 32499 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.066s
I20240426 13:22:19.451633 31862 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.452389 25102 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 26.
I20240426 13:22:19.482076 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.482949 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 3613) took 1019 ms. Trace:
I20240426 13:22:19.448633 32493 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:19.378899 32689 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.465169 32694 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:19.491796 32726 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:19.463989 32732 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20240426 13:22:19.490552 24947 rpcz_store.cc:276] 0426 13:22:18.463499 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:18.978441 (+514942us) service_pool.cc:225] Handling call
0426 13:22:18.978705 (+   264us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:18.978888 (+   183us) raft_consensus.cc:1520] Early marking committed up to index 13
0426 13:22:18.996126 (+ 17238us) log.cc:842] Serialized 14 byte log entry
0426 13:22:19.479305 (+483179us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:22:19.479375 (+    70us) raft_consensus.cc:1631] Marking committed up to 13
0426 13:22:19.480706 (+  1331us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:19.481562 (+   856us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:19.482882 (+  1320us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":18206,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":4283,"raft.queue_time_us":549,"thread_start_us":482211,"threads_started":1}
I20240426 13:22:19.379606 32357 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:19.517980 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 416952us
I20240426 13:22:19.507755 32735 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:19.527537 32735 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:19.513254 32732 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.529506 32732 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:19.538178 32707 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.540058 32747 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.528581 32735 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:19.569118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 329859us
I20240426 13:22:19.575904 32628 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:19.576025 32659 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.576654 32628 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:19.577728 32628 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.578353 32659 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.579056 32628 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
W20240426 13:22:19.591956 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 353631us
I20240426 13:22:19.642930 24852 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.571564 32689 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.644850 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.572892 32707 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.667105 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.683480 31862 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.687053 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:19.661875   363 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20240426 13:22:19.695657 32671 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:19.699764 32339 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:19.713250 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:19.713869 25620 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:19.714504 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 232640us
W20240426 13:22:19.715510 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 196552us
W20240426 13:22:19.720399 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 201413us
I20240426 13:22:19.723901 25098 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:19.728353 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.723505 24995 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
I20240426 13:22:19.728919 25022 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 20
I20240426 13:22:19.696625 32610 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.729024 24995 leader_election.cc:401] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20240426 13:22:19.733072 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.707628 32747 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:19.739890 25022 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:19.742235 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:19.746405 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:19.750885 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.751808 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:19.754859 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:19.758136   371 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.247s	user 0.000s	sys 0.004s
W20240426 13:22:19.760670   375 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20240426 13:22:19.750489 32659 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:19.751108 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:19.764474 25097 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:19.770295 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.741922 25620 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 26.
I20240426 13:22:19.772548 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.773903 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:19.774299 24956 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20240426 13:22:19.743712 32339 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:19.785112 32747 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240426 13:22:19.785732 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.777880 32659 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:19.791576   372 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20240426 13:22:19.790066 32493 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20240426 13:22:19.798846   373 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20240426 13:22:19.789211   390 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25.
W20240426 13:22:19.863762 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 176789us
W20240426 13:22:19.865190 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 205842us
I20240426 13:22:19.867877 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.806389 32707 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:19.865366 25097 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
W20240426 13:22:19.876617 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 128573us
W20240426 13:22:19.877380 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 129934us
W20240426 13:22:19.878191 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119088us
I20240426 13:22:19.879034 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.807663   389 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20240426 13:22:19.881629 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:19.885335   382 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:19.866942 32689 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.885381 32339 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.886350 32689 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.888393 32689 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.879763   387 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.889468 32610 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.889768   387 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.860721 32756 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20240426 13:22:19.878594 32493 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.893440 32493 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:22:19.895478 32493 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:19.896458 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.896533 32707 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:19.897687 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 23 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:19.898913 32707 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.898063 32563 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:19.900487 32563 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:19.901284 32563 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.899339 24852 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 7 to 22. New cstate: current_term: 22 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.902123 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:19.902565 32563 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
W20240426 13:22:19.902879 32225 consensus_meta.cc:287] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.589s	user 0.005s	sys 0.005s
I20240426 13:22:19.902645 24852 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 23, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.903306 32225 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.904377 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.904999 25099 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:19.905398 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:19.897727 24849 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 281) took 73510 ms (client timeout 15000 ms). Trace:
I20240426 13:22:19.906214 32610 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:19.907757 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.907909 32659 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.891870   387 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.910589 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.912364 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:19.904269 24852 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:19.919097 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:19.920074 25024 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20240426 13:22:19.922607 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 19 to 24, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.923480 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.927307 25025 raft_consensus.cc:2354] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:19.928032 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:19.928607 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.930862 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:19.931604 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:19.935417 25023 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:19.932451 25097 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
W20240426 13:22:19.944206   394 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20240426 13:22:19.936888 24852 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 19, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.947655 24955 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 24. Current term is 25. Ops: [24.12-24.12]
W20240426 13:22:19.948650   369 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.485s	user 0.002s	sys 0.000s
I20240426 13:22:19.949201 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 16. Current term is 17. Ops: []
I20240426 13:22:19.950991 32726 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.947640 24852 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:19.951470 32726 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:19.919252   392 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:19.922907 32493 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:19.952701 32493 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:19.952288 24852 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 13 to 14, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 14 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.953085 32493 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:19.957818   313 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.958492   313 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.958930 32732 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:19.962363 32225 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:19.947571 32676 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:19.966260 32518 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.966948 32518 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:19.953707 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 17, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 17 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:19.968386 24852 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:19.968950 32518 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:19.970762 32714 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:19.971573 32675 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:19.974261 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:19.976372 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:19.993348 32735 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:19.993891 32735 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:22:19.980865 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57456.4us
W20240426 13:22:19.984844 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91886.6us
I20240426 13:22:19.927240   387 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:20.004655   313 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:20.006752 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53493.4us
I20240426 13:22:20.010471 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:20.008618 24921 leader_election.cc:398] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:19.994309 32735 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:20.023092 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:22:19.906296 24849 rpcz_store.cc:269] 0426 13:21:06.387764 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:06.389769 (+  2005us) service_pool.cc:225] Handling call
0426 13:22:19.897628 (+73507859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:13.808702 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:13.809367 (+   665us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:13.809440 (+    73us) write_op.cc:427] Acquired schema lock
0426 13:22:13.809476 (+    36us) tablet.cc:660] Decoding operations
0426 13:22:13.810197 (+   721us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:13.810283 (+    86us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:13.810314 (+    31us) tablet.cc:683] Acquiring locks for 4 operations
0426 13:22:13.810814 (+   500us) tablet.cc:699] Row locks acquired
0426 13:22:13.810840 (+    26us) write_op.cc:260] PREPARE: finished
0426 13:22:13.811000 (+   160us) write_op.cc:270] Start()
0426 13:22:13.811136 (+   136us) write_op.cc:275] Timestamp: P: 1714137733810935 usec, L: 0
0426 13:22:13.811167 (+    31us) op_driver.cc:331] REPLICATION: starting
0426 13:22:13.811823 (+   656us) log.cc:842] Serialized 1677 byte log entry
0426 13:22:14.399850 (+588027us) op_driver.cc:444] REPLICATION: finished
0426 13:22:14.652433 (+252583us) write_op.cc:300] APPLY: starting
0426 13:22:14.653877 (+  1444us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:14.739057 (+ 85180us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:14.739237 (+   180us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:14.741954 (+  2717us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:14.742328 (+   374us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=8,mrs_lookups=0
0426 13:22:14.742408 (+    80us) write_op.cc:311] APPLY: finished
0426 13:22:14.742770 (+   362us) log.cc:842] Serialized 49 byte log entry
0426 13:22:14.742968 (+   198us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:14.743107 (+   139us) write_op.cc:446] Released schema lock
0426 13:22:14.743182 (+    75us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:14.836122 (+ 92940us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:14.839827 (+  3705us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":192646,"reported_tablets":50,"spinlock_wait_cycles":171264,"child_traces":[["op",{"apply.queue_time_us":252273,"apply.run_cpu_time_us":9122,"apply.run_wall_time_us":212429,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20855,"num_ops":4,"prepare.queue_time_us":801910,"prepare.run_cpu_time_us":49305,"prepare.run_wall_time_us":266059,"raft.queue_time_us":220979,"raft.run_cpu_time_us":30018,"raft.run_wall_time_us":222914,"replication_time_us":588579,"thread_start_us":942727,"threads_started":4,"wal-append.queue_time_us":343165,"wal-append.run_cpu_time_us":27881,"wal-append.run_wall_time_us":1641193}]]}
I20240426 13:22:20.045670 32676 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:22:20.046212 32732 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.046720   380 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:20.010183   400 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20240426 13:22:20.053762   305 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:20.055171 25028 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20240426 13:22:20.055819 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:20.058382 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47596us
I20240426 13:22:20.063673 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20240426 13:22:20.066242 32726 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.114547 25025 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
I20240426 13:22:20.069800 24995 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:19.995183 32714 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.088596 25099 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 12. (term mismatch)
I20240426 13:22:20.121281 25099 pending_rounds.cc:76] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 11
I20240426 13:22:20.121611 25099 pending_rounds.cc:97] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20240426 13:22:20.121934 25099 raft_consensus.cc:2866] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:20.089461 25102 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20240426 13:22:20.089483 24956 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 25. Current term is 26. Ops: []
I20240426 13:22:20.090333 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:20.099428 32676 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Stepping down as leader of term 16
I20240426 13:22:20.109452 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:20.109794 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:20.112475   305 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.119709 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:20.120476 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50779.5us
I20240426 13:22:20.128436 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:20.136118 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:20.137681 24995 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:20.138648 24995 leader_election.cc:401] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20240426 13:22:20.141881 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.144891 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.150646 25097 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 17: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:20.131240 32563 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:20.152818 25031 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:20.126874   400 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:20.127434 32676 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:20.145431   389 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:20.129490   403 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:20.195611 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.187449 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.192395 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:20.192801 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:20.198406 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53587.8us
I20240426 13:22:20.199167 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.199317 25098 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:20.202240 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:20.202977 24948 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.203217 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:20.204313 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:20.206212 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:20.206873 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 17.
I20240426 13:22:20.208578 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:20.129271 32732 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.063s
I20240426 13:22:20.210233 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.210595 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:20.204216 32628 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:20.147604 32736 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:20.149169   387 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:20.146863   404 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:20.145004 32627 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:20.148895 32357 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:20.141600 32747 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20240426 13:22:20.188958   305 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:20.212136 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:20.217751 25620 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20240426 13:22:20.212852 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:20.213853 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:20.298408 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.200891 32563 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:20.148209 32493 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.190027 32726 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.183360 32735 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.149307   401 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.128s
I20240426 13:22:20.208084   389 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.182387 32766 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.106s
I20240426 13:22:20.181200 32339 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.109s
I20240426 13:22:20.326959 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.385624 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:20.306365 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.185088 32714 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.329417 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.207324 32707 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader election won for term 17
I20240426 13:22:20.209162   399 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:20.393678 24952 raft_consensus.cc:2354] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:22:20.209475   388 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20240426 13:22:20.395716 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 181918us
I20240426 13:22:20.274902   400 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20240426 13:22:20.312639   411 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20240426 13:22:20.212081   390 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:20.406708 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:20.338186   406 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20240426 13:22:20.419560 24950 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:20.323633 32676 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.324087 32357 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.210655   403 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.460930 25024 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:20.458879 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:20.211506 32305 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:20.208984 31862 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:20.500561 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:20.482388   403 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:20.489176   303 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:20.493312 32735 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.495378 32493 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.471655   402 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:20.497869 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:20.299250 32689 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.134s
I20240426 13:22:20.517380 25103 raft_consensus.cc:2354] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:20.500061 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:20.517117   402 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.518060 25022 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 22: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:20.435264 32726 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:20.518612 24921 leader_election.cc:398] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:20.518990 24921 leader_election.cc:401] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20240426 13:22:20.519240   402 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.449486   389 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:20.419639   305 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:20.214421   391 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.206s
I20240426 13:22:20.441248   390 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:20.530382 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:20.532769 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:20.443814 32736 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26.
I20240426 13:22:20.530565 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.535276 32708 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:20.537757 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:20.539651 25022 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20240426 13:22:20.541066 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.542071 32659 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:20.426213 32707 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:20.543188 24950 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:20.544438 32659 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:20.544929 32707 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.547399 32726 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 23. Current term is 24.
I20240426 13:22:20.549733 25105 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:20.553112 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.554517 32659 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:20.555469 32659 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:20.557006 32659 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.564289 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:20.565380 25103 raft_consensus.cc:2396] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20240426 13:22:20.397861   382 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.082s
I20240426 13:22:20.519953 32357 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:20.397087   387 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.572100 32676 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 16 FOLLOWER]: Advancing to term 17
I20240426 13:22:20.572502   390 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.573556   387 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:20.530073   305 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:20.579500   305 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:20.508390 32563 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.581682 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:20.584100 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 151387us
W20240426 13:22:20.567731 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 244618us
I20240426 13:22:20.590307 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:20.590801 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:20.590919 25620 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:20.606298 25620 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20240426 13:22:20.583503 32357 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:20.619318 24949 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:20.620072 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:20.620985 25622 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 21.
I20240426 13:22:20.624552 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.567051 32628 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20240426 13:22:20.627830 24947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6277899576264e68aa43514e5b5c676e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:20.634584 24947 raft_consensus.cc:595] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.634934 24947 raft_consensus.cc:603] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:20.630931 25026 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20240426 13:22:20.645922 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 222373us
I20240426 13:22:20.647163 24947 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 25. Current term is 26. Ops: []
I20240426 13:22:20.647316 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.654560 25103 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:20.653055 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.664144 24995 leader_election.cc:398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:22:20.665374 24995 leader_election.cc:401] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:20.664762 31862 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:20.667730 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.667903   419 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:20.635922 32357 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.694603 32732 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20240426 13:22:20.649216   387 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20240426 13:22:20.696938 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:20.703886 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
W20240426 13:22:20.704449   417 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20240426 13:22:20.707751 24955 raft_consensus.cc:2396] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20240426 13:22:20.687956   399 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:22:20.688023   419 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:20.684334   390 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22.
I20240426 13:22:20.708642 24951 raft_consensus.cc:595] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.720290 24951 raft_consensus.cc:603] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:20.712467   399 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:20.713277 24949 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 24. Current term is 25. Ops: []
I20240426 13:22:20.715791 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:20.663861   305 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.706595 32732 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:20.707338 32736 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:20.704476   380 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:22:20.741966 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72192.6us
I20240426 13:22:20.717864   419 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:20.782302   305 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:20.743018 32732 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:20.766060   380 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.787009   380 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:20.768538 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 74898.9us
I20240426 13:22:20.785369 32732 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:20.713670 32493 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20240426 13:22:20.789942 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.790580 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:20.800465 25027 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:20.825665 25027 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
I20240426 13:22:20.840642 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:20.841643 25097 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:20.845373 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:20.858942 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:20.855305 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:20.860503 25622 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:20.846907 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:20.862524 25621 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 23: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:20.848521 32700 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
W20240426 13:22:20.866842 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56624.1us
I20240426 13:22:20.857081 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:20.881675 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:20.888612   426 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20240426 13:22:20.889210 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:20.892256 32305 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:20.842411 32357 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:20.829844 32526 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20240426 13:22:20.897862 32526 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:20.898226 32526 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:20.898991 32526 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.905076   313 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:20.905433 24954 raft_consensus.cc:1203] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Deduplicated request from leader. Original: 24.19->[25.20-25.20]   Dedup: 25.20->[]
I20240426 13:22:20.905651   313 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:20.906040 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20240426 13:22:20.906198   313 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:20.873113 32567 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:20.908612 32567 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:20.909435 32567 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.872551 32726 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:20.912427 32726 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:20.912220   435 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 20, Time since last communication: 0.001s
I20240426 13:22:20.912915 32726 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:20.913262   435 consensus_queue.cc:1225] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 26.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20240426 13:22:20.918869   392 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:20.919338   392 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:20.920156   392 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.921375   392 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:20.924173   399 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:20.925580   399 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:20.932633 25624 raft_consensus.cc:1203] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Deduplicated request from leader. Original: 24.19->[25.20-25.20]   Dedup: 25.20->[]
I20240426 13:22:20.933295 25624 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20240426 13:22:20.934402 32526 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:20.935262 32628 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 20, Time since last communication: 0.000s
W20240426 13:22:20.942521   421 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.402s	user 0.009s	sys 0.000s
I20240426 13:22:20.943876 25022 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:22:20.944012 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:20.944460 25022 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:20.944761 24951 raft_consensus.cc:595] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:20.945183 24951 raft_consensus.cc:603] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:20.945457 25022 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:20.947182 25022 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:20.975883 32610 raft_consensus.cc:979] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:20.976487 32610 raft_consensus.cc:1067] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:20.987118 24955 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:20.996415 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:21.001317 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:21.001986 24956 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:21.002763   436 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:21.002513   313 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.025501 25022 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 20. (term mismatch)
I20240426 13:22:21.026032 25022 pending_rounds.cc:76] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 19
I20240426 13:22:21.029536 25022 pending_rounds.cc:97] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20240426 13:22:21.029861 25022 raft_consensus.cc:2866] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:21.046880 24956 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20240426 13:22:20.994977 32563 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:20.907814 32732 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.100330 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:22:21.107770 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46245.6us
W20240426 13:22:21.054601   414 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20240426 13:22:21.140192 25105 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58866
I20240426 13:22:21.141775 25105 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:21.142146 25105 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:21.057089 32736 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:21.145732 24951 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:21.062321 32689 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.082s
W20240426 13:22:21.162374 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60197.6us
I20240426 13:22:21.102049 32628 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20240426 13:22:21.171357 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:21.147936   313 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:21.171981 25620 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:21.044494   419 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.173564 25105 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.190524 25024 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:21.186400 32339 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20240426 13:22:21.191744 32726 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.193897 32726 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:21.195396 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:21.197805 24955 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:21.201300 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:21.211232 25105 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:21.210964 24955 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:21.215574 32732 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:21.216554 24947 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 21. Current term is 22. Ops: []
I20240426 13:22:21.217533 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:21.218605 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:22:21.238253 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 61544.8us
I20240426 13:22:21.238360 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:21.215807 32689 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20240426 13:22:21.239272 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:21.222388 25023 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:21.239883 32689 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:21.240365 32689 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:21.229633   419 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:21.241144 32689 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.242303 32689 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:21.243985 25104 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20240426 13:22:21.244239 25097 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 17. Current term is 18. Ops: []
I20240426 13:22:21.245242 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:21.251061 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:21.241539   313 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
W20240426 13:22:21.260928 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60135.2us
W20240426 13:22:21.233929   438 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20240426 13:22:21.256903 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:21.278048 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:21.280144 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:21.290666 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:21.295740 25023 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:21.292668 32659 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240426 13:22:21.298761   387 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20240426 13:22:21.276782   443 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20240426 13:22:21.286089   441 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.070s	user 0.000s	sys 0.002s
I20240426 13:22:21.319032   313 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:21.322119   313 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.324728   419 raft_consensus.cc:979] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:21.325209   419 raft_consensus.cc:1067] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:21.328099 32493 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:21.328611 32493 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.330205 25098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58866
I20240426 13:22:21.330716 25098 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:21.331995 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:21.332644 25098 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:21.298305 32732 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20240426 13:22:21.336774 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:21.337453 25031 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:21.339282 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:21.330569 32493 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:21.369154 24949 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Stepping down as leader of term 26
I20240426 13:22:21.369587 24949 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:21.370525 24949 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.362812 25098 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.350261 32707 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:21.430940 25098 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:21.366817 32747 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20240426 13:22:21.386731 32707 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:21.444818 25103 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 26. Current term is 27. Ops: []
I20240426 13:22:21.508744 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:21.509722 24954 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:22:21.510318 24954 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:21.510900 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82406.7us
I20240426 13:22:21.516201 24954 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.517297 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:21.518646 25024 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:21.523967 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82373us
I20240426 13:22:21.524127 32735 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:21.524674 32735 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.526630 32735 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:21.514279 32659 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:21.533938 32659 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:21.560767 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 114752us
I20240426 13:22:21.443063 32707 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:21.557099 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:21.565899 25103 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 17.
I20240426 13:22:21.567387 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:21.580140 25024 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
W20240426 13:22:21.580916   446 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20240426 13:22:21.583364 32735 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20240426 13:22:21.600411 32305 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:21.641884 32659 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:21.648627   447 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20240426 13:22:21.644891 32735 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:22:21.670338   439 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.627s	user 0.004s	sys 0.000s
W20240426 13:22:21.677984   453 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20240426 13:22:21.725495 24955 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20240426 13:22:21.726627 24955 pending_rounds.cc:76] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 13
I20240426 13:22:21.726967 24955 pending_rounds.cc:97] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20240426 13:22:21.727380 24955 raft_consensus.cc:2866] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:21.733060 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 1129) took 1587 ms. Trace:
I20240426 13:22:21.733943 25026 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:22:21.745882 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 3100) took 1034 ms. Trace:
I20240426 13:22:21.746374 24947 rpcz_store.cc:276] 0426 13:22:20.711307 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:20.711969 (+   662us) service_pool.cc:225] Handling call
0426 13:22:20.712234 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:22:21.216967 (+504733us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:21.745282 (+528315us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:21.745647 (+   365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1151671552}
I20240426 13:22:21.748402 24949 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:21.668641 32735 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:21.733741 25025 rpcz_store.cc:276] 0426 13:22:20.145451 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:20.146458 (+  1007us) service_pool.cc:225] Handling call
0426 13:22:20.146756 (+   298us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:22:20.655632 (+508876us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:22:20.655674 (+    42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:22:20.850385 (+194711us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b2c00011698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:22:20.851001 (+   616us) log.cc:842] Serialized 52 byte log entry
0426 13:22:21.020699 (+169698us) raft_consensus.cc:1631] Marking committed up to 15
0426 13:22:21.054426 (+ 33727us) log.cc:842] Serialized 14 byte log entry
0426 13:22:21.056298 (+  1872us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:22:21.109660 (+ 53362us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54004169d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:22:21.109735 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:22:21.192193 (+ 82458us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b18003d3600. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:22:21.543583 (+351390us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:22:21.732306 (+188723us) raft_consensus.cc:1671] finished
0426 13:22:21.732390 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:22:21.732759 (+   369us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33608,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":4581,"raft.queue_time_us":7455,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":50986,"spinlock_wait_cycles":1132241408,"thread_start_us":344270,"threads_started":2,"wal-append.queue_time_us":191852}
W20240426 13:22:21.777711 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 163824us
I20240426 13:22:21.652688 32305 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.790228 32700 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:21.791280 32700 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.818217 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:21.819685 24956 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:21.820192 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:21.821007 25102 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 25. Current term is 26. Ops: []
I20240426 13:22:21.815691 25620 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20240426 13:22:21.821228 25099 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 26.
I20240426 13:22:21.825656 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:21.827214 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:21.828147 25025 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 16.
I20240426 13:22:21.829861   419 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20240426 13:22:21.849421 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66723.2us
I20240426 13:22:21.849886 32305 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:21.853222 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:21.860289 32700 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:21.861039 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:21.867832 24949 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 27.
I20240426 13:22:21.878363   416 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:21.879734   387 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:22:21.880406   387 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:21.880797   387 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:21.868368   462 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20240426 13:22:21.893072   396 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:21.893188   301 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:21.893932   396 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:21.897294 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:21.904325 24954 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:21.915378   403 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:21.918483   403 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:21.918973   403 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:21.895498   396 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:21.936511   403 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:21.938480   403 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:21.959288 24954 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
W20240426 13:22:21.954663 25023 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.658s	user 0.011s	sys 0.014s
I20240426 13:22:21.961287 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:21.960603 25023 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 27.
I20240426 13:22:21.976445 32225 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:21.977120 32225 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:21.978595 32225 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:22.012943 32736 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:22.013672 32736 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.029907 25023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:22.033727 25023 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:22.032927 32736 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.048569 32225 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:22.060379 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:22:22.039009 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:22.034072 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:22.064507 32339 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:22.065200 32339 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:22.066411   397 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:22.066318 32339 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.068830 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:22:22.060446 32563 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.960s	user 0.011s	sys 0.010s
I20240426 13:22:22.085654 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:22.088976 25097 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20240426 13:22:22.102480 32735 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.105058 32735 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.105634 25102 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:22.106838 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:22.107851 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:22.100145   387 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.124056 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:22.124724 25023 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 16 FOLLOWER]: Advancing to term 18
I20240426 13:22:22.127840 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:22.133185 25099 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:22.136802 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:22.131695   387 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.141162 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:22.141927 25098 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:22.143362 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:22.144450 32628 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20240426 13:22:22.161002 32707 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.575s	user 0.005s	sys 0.009s
I20240426 13:22:22.161629 25098 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 27.
I20240426 13:22:22.161520 32707 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.162815 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:22.163682 32707 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.146739 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:22.163980 32736 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:22.164814   387 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:22.168136 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:22.168265 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:22.175757 25030 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:22.182049 24947 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20240426 13:22:22.179325 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:22.182089 32736 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.184182   387 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:22.185880 32736 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.187688 25099 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:22.177672 32628 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:22.191089 32628 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:22.191601 32628 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:22.193694 32627 raft_consensus.cc:979] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:22.195034   403 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:22.200215 32339 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20240426 13:22:22.185276   387 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.248139 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:22.249025 25102 raft_consensus.cc:2354] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 26. Current term is 27.
I20240426 13:22:22.250419 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:22.242332 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:22.242130 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:22.241111 32627 raft_consensus.cc:1067] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:22.262323 32735 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:22.265264   397 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:22.274135 32735 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:22.274235   397 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.275180 32735 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.276832 32707 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:22.277442 32707 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.279446 32707 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.281215   397 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:22.282335 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:22.283001 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:22.282974 24956 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 22.
I20240426 13:22:22.286381 24996 leader_election.cc:398] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 26. Current term is 27.
I20240426 13:22:22.286715 24996 leader_election.cc:401] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:22.287091 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:22.265645 25030 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 26.
I20240426 13:22:22.287941 32563 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:22.288182 32225 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 27
I20240426 13:22:22.288563 32563 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:22.289014 32563 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:22.292869 32628 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.295341 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:22.308130 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:22.317322 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:22.321066 32747 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:22.318229   301 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:22.323323 25098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58860
I20240426 13:22:22.331154 25098 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:22.333608 25098 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:22.343084 32628 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.346319 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:22.346282 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:22.349892 24947 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:22.354575   403 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:22.364009 25098 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:22.375916 25098 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:22.378082 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:22.379927 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:22.380551 25023 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:22.380892 25023 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:22.388628 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:22.391660 25620 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:22.377643   403 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.386684 25023 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:22.397264 32671 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:22.397881 32671 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.403692 25023 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:22.404917 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:22.405634 25031 raft_consensus.cc:2354] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 25. Current term is 26.
I20240426 13:22:22.409446 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:22.416819 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:22.417608 25070 leader_election.cc:398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 25. Current term is 26.
I20240426 13:22:22.411458 32225 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 26. Current term is 27.
I20240426 13:22:22.419135 25070 leader_election.cc:401] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:22.399958 32671 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:22.408550   475 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20240426 13:22:22.424068 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:22.425948 24947 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:22.428730   403 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:22.438366 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:22.429530 32671 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 26
I20240426 13:22:22.445206 25102 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:22.447559 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:22.449388 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:22.433537 32563 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.453375 32627 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:22.475903 25620 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:22.451792 32225 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:22.479511 25097 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 23: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:22.479960 25023 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:22.481504 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:22.489123   301 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:22.487396 32225 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:22.494232 32747 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:22:22.499634 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96043.4us
I20240426 13:22:22.502969 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:22.505756 32747 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.508879 32493 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:22.511049 25023 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20240426 13:22:22.518806   396 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:22.502661   301 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:22.520119   301 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:22.523756 24853 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 14 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.525485 24853 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.532442 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:22.535271 24853 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 19. New cstate: current_term: 19 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:22.539319 32563 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:22.539654 24853 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.541512 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:22.541319 24853 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:22.543341 24951 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20240426 13:22:22.547600 25023 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20240426 13:22:22.548605   397 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:22.511550 32225 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.543053 24853 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.558189 32700 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:22.548633 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 181) took 10589 ms. Trace:
I20240426 13:22:22.568894 32493 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:22.590960 32671 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 25. Current term is 26.
I20240426 13:22:22.591747 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 19, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 19 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:22.596447 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.604374 24853 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 15, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 15 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.610826 32627 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:22.611558 25022 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:22.612151 24853 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 14 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 19 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.611953 24947 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:22.652722 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:22.653566 32700 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:22.654047 32700 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:22.654999 32700 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:22.619175 25022 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:22.625931 24951 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20240426 13:22:22.562466 24852 rpcz_store.cc:276] 0426 13:22:11.959192 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:11.963199 (+  4007us) service_pool.cc:225] Handling call
0426 13:22:22.548451 (+10585252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:20.668492 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:20.806161 (+137669us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:20.808295 (+  2134us) write_op.cc:427] Acquired schema lock
0426 13:22:20.810736 (+  2441us) tablet.cc:660] Decoding operations
0426 13:22:20.884500 (+ 73764us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:20.894140 (+  9640us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:20.894174 (+    34us) tablet.cc:683] Acquiring locks for 19 operations
0426 13:22:20.895704 (+  1530us) tablet.cc:699] Row locks acquired
0426 13:22:20.895732 (+    28us) write_op.cc:260] PREPARE: finished
0426 13:22:20.895900 (+   168us) write_op.cc:270] Start()
0426 13:22:20.896013 (+   113us) write_op.cc:275] Timestamp: P: 1714137740895847 usec, L: 0
0426 13:22:20.896056 (+    43us) op_driver.cc:331] REPLICATION: starting
0426 13:22:20.896711 (+   655us) log.cc:842] Serialized 7378 byte log entry
0426 13:22:21.650963 (+754252us) op_driver.cc:444] REPLICATION: finished
0426 13:22:21.678026 (+ 27063us) write_op.cc:300] APPLY: starting
0426 13:22:21.678114 (+    88us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:21.688497 (+ 10383us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:21.688591 (+    94us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:22.186205 (+497614us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:22.186725 (+   520us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0426 13:22:22.186805 (+    80us) write_op.cc:311] APPLY: finished
0426 13:22:22.189468 (+  2663us) log.cc:842] Serialized 172 byte log entry
0426 13:22:22.191314 (+  1846us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:22.191593 (+   279us) write_op.cc:446] Released schema lock
0426 13:22:22.191667 (+    74us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:22.200736 (+  9069us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:22.200767 (+    31us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2099155,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":26744,"apply.run_cpu_time_us":29383,"apply.run_wall_time_us":523254,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83585,"num_ops":19,"prepare.queue_time_us":357411,"prepare.run_cpu_time_us":84105,"prepare.run_wall_time_us":530368,"raft.queue_time_us":255159,"raft.run_cpu_time_us":6787,"raft.run_wall_time_us":20202,"replication_time_us":754787,"thread_start_us":677095,"threads_started":4,"wal-append.queue_time_us":318943}]]}
I20240426 13:22:22.651861 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.659040 24853 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 19, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 19 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.660001   397 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:22.676283 24853 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 10 to 15, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 15 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:22.683799 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 17, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 17 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.688565 24853 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 20. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:22.690286 24853 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 17, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 17 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.692909 24853 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.704329 24853 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.709174 25022 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20240426 13:22:22.711174 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 9 to 12, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 12 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } }
I20240426 13:22:22.713225   419 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20240426 13:22:22.726274 24853 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 20. New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.727692 24853 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 6 to 17, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 17 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.729756 24853 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 16, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 16 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.736858 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.766296 24853 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 18, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:22.774255 24853 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.775800 24853 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 16. New cstate: current_term: 16 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:22.793176 25097 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20240426 13:22:22.794675   419 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:22.830013 25022 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:22:22.831374   419 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:22:22.852483 25099 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20240426 13:22:22.854099 32735 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240426 13:22:22.948249 24947 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20240426 13:22:22.949774   403 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.003s
I20240426 13:22:22.959975 25098 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20240426 13:22:22.961498 32225 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20240426 13:22:23.020159 25022 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20240426 13:22:23.023536   419 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:23.034066 25097 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20240426 13:22:23.043069   419 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:23.087980 25620 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20240426 13:22:23.089743 32563 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:23.098628 25022 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20240426 13:22:23.101327 32700 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:23.109095 25023 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:23.111212 32700 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:23.130368 32735 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:23.130996 32735 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.133056 32735 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:23.136984 32714 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:23.137670 32714 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.141103 32714 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:23.143694 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:23.144794 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:23.146346 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:23.148262 32730 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:23.148741 32730 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:23.149175 32730 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:23.172746 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:23.173548 25030 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
W20240426 13:22:23.175069 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63870.4us
I20240426 13:22:23.176381 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:23.181071   419 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:23.181563   419 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:23.181921   419 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.188766 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:23.193564 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:23.194557 24951 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:23.231918 24947 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:23.241281 32730 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.244179 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:23.245024 24947 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 22: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:22:23.293414 32671 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:22:23.331902 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 277) took 32442 ms (client timeout 15000 ms). Trace:
I20240426 13:22:23.358247 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:23.359982 32730 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:23.360875 24848 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 22, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.362540 24848 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 16 to 24. New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.356660   419 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.365254   419 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:23.332415 24853 rpcz_store.cc:269] 0426 13:21:50.889476 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:50.889711 (+   235us) service_pool.cc:225] Handling call
0426 13:22:23.331788 (+32442077us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:22.827386 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:22.832467 (+  5081us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:22.832544 (+    77us) write_op.cc:427] Acquired schema lock
0426 13:22:22.832563 (+    19us) tablet.cc:660] Decoding operations
0426 13:22:22.834418 (+  1855us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:22.834526 (+   108us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:22.834559 (+    33us) tablet.cc:683] Acquiring locks for 26 operations
0426 13:22:22.863702 (+ 29143us) tablet.cc:699] Row locks acquired
0426 13:22:22.863741 (+    39us) write_op.cc:260] PREPARE: finished
0426 13:22:22.863918 (+   177us) write_op.cc:270] Start()
0426 13:22:22.864103 (+   185us) write_op.cc:275] Timestamp: P: 1714137742863840 usec, L: 0
0426 13:22:22.864132 (+    29us) op_driver.cc:331] REPLICATION: starting
0426 13:22:22.864959 (+   827us) log.cc:842] Serialized 10072 byte log entry
0426 13:22:22.911341 (+ 46382us) op_driver.cc:444] REPLICATION: finished
0426 13:22:22.988332 (+ 76991us) write_op.cc:300] APPLY: starting
0426 13:22:22.988449 (+   117us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:23.019547 (+ 31098us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:23.019651 (+   104us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:23.063804 (+ 44153us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:23.075255 (+ 11451us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=52,mrs_lookups=0
0426 13:22:23.095193 (+ 19938us) write_op.cc:311] APPLY: finished
0426 13:22:23.104483 (+  9290us) log.cc:842] Serialized 228 byte log entry
0426 13:22:23.104663 (+   180us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:23.104859 (+   196us) write_op.cc:446] Released schema lock
0426 13:22:23.104921 (+    62us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:23.113856 (+  8935us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:23.113895 (+    39us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":22678,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":76218,"apply.run_cpu_time_us":30182,"apply.run_wall_time_us":126249,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112859,"num_ops":26,"prepare.queue_time_us":25837,"prepare.run_cpu_time_us":6841,"prepare.run_wall_time_us":38118,"replication_time_us":47106,"thread_start_us":81376,"threads_started":2}]]}
I20240426 13:22:23.365912 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:23.366506 25023 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.366737 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:23.367218 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:23.367929 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:23.370023   461 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:23.373297 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:23.363942 24848 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:23.370523   461 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:23.376960 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:23.378585 32735 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:23.376310 24848 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 23, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.378808   461 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.395081 32735 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:23.398036 32735 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.396766 24848 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.406446 24848 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:23.408489 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66321.6us
I20240426 13:22:23.410732 24848 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:23.429553 32730 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:23.429174 24848 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.430127 32730 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.432878 32730 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:23.433221 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:22:23.435988 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:23.436743 25022 raft_consensus.cc:2354] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23.
I20240426 13:22:23.440780 25589 leader_election.cc:398] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23.
I20240426 13:22:23.441124 25589 leader_election.cc:401] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:22:23.441773 32714 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.456223   419 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:23.467826 24848 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.481555   419 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.492319 24848 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.501147 24848 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 19 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.506553 24848 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 17, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 17 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.511837 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:23.515450   419 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:23.495081 32714 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23.
I20240426 13:22:23.508101 24848 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 17 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.525004 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:23.525636 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:23.526767 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:23.527983 32735 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:23.528451 32735 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:23.528824 32735 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.529587 24848 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 23. New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.535943 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:23.548923 24848 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 17 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.550073 32735 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.553143 24848 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.555630 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:23.556195 25024 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.556138 32735 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:23.556236 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:23.555960 24848 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 21 to 22, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 22 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.559828 24848 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.562539 24848 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.564193 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:23.563827 24848 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 7 to 15, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 15 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.565204 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:23.565976 32735 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:23.566475 32735 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:23.567232 32735 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.565274 24848 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 12 to 13, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 13 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } }
W20240426 13:22:23.616437 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44963.3us
I20240426 13:22:23.632670 24848 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.639787 24848 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:23.655719 24848 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 15 to 17, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 17 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:23.666266 24848 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 11 to 20. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:23.676841 25024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:23.677500 25024 raft_consensus.cc:595] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.677855 25024 raft_consensus.cc:603] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.682313 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:23.682826 24951 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.683079 24951 raft_consensus.cc:603] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.687009 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:23.687531 24951 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.687779 24951 raft_consensus.cc:603] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.691344 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:23.691910 24951 raft_consensus.cc:595] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.692282 24951 raft_consensus.cc:603] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.700493 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:23.701082 24951 raft_consensus.cc:595] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.701444 24951 raft_consensus.cc:603] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.706784 24951 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:23.714672 24951 raft_consensus.cc:595] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:23.718806 24951 raft_consensus.cc:603] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:23.727749 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:22:23.729717 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:23.737704 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:23.807765 24951 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:23.809415 32714 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:23.810161 32714 consensus_queue.cc:1225] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 23.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20240426 13:22:23.810657 32714 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:23.810974 32714 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:23.811821 32714 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:23.813894 32714 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:23.875813 25022 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20240426 13:22:23.880681 32735 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:23.993155 25621 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20240426 13:22:23.994988 32357 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:23.997581 24846 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 12 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.029274 24846 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 15 to 17, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 17 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.031213 25620 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20240426 13:22:24.031680 25620 pending_rounds.cc:76] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 15
I20240426 13:22:24.031924 25620 pending_rounds.cc:97] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20240426 13:22:24.032244 25620 raft_consensus.cc:2866] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:22:24.033886 32735 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240426 13:22:24.038414 24848 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 278) took 11490 ms (client timeout 15000 ms). Trace:
W20240426 13:22:24.038877 24848 rpcz_store.cc:269] 0426 13:22:12.548400 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:12.600622 (+ 52222us) service_pool.cc:225] Handling call
0426 13:22:24.038348 (+11437726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:23.704910 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:23.712195 (+  7285us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:23.712241 (+    46us) write_op.cc:427] Acquired schema lock
0426 13:22:23.712263 (+    22us) tablet.cc:660] Decoding operations
0426 13:22:23.720403 (+  8140us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:23.720500 (+    97us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:23.720539 (+    39us) tablet.cc:683] Acquiring locks for 25 operations
0426 13:22:23.722690 (+  2151us) tablet.cc:699] Row locks acquired
0426 13:22:23.722712 (+    22us) write_op.cc:260] PREPARE: finished
0426 13:22:23.722939 (+   227us) write_op.cc:270] Start()
0426 13:22:23.723058 (+   119us) write_op.cc:275] Timestamp: P: 1714137743722880 usec, L: 0
0426 13:22:23.723077 (+    19us) op_driver.cc:331] REPLICATION: starting
0426 13:22:23.723811 (+   734us) log.cc:842] Serialized 9695 byte log entry
0426 13:22:23.779889 (+ 56078us) op_driver.cc:444] REPLICATION: finished
0426 13:22:23.812439 (+ 32550us) write_op.cc:300] APPLY: starting
0426 13:22:23.812525 (+    86us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:23.920986 (+108461us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:23.921083 (+    97us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:23.942734 (+ 21651us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:23.943062 (+   328us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=50,mrs_lookups=0
0426 13:22:23.943105 (+    43us) write_op.cc:311] APPLY: finished
0426 13:22:23.943635 (+   530us) log.cc:842] Serialized 220 byte log entry
0426 13:22:23.944365 (+   730us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:23.944564 (+   199us) write_op.cc:446] Released schema lock
0426 13:22:23.944621 (+    57us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:23.951549 (+  6928us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:23.951584 (+    35us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":32246,"apply.run_cpu_time_us":28400,"apply.run_wall_time_us":139857,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108677,"num_ops":25,"prepare.queue_time_us":12237,"prepare.run_cpu_time_us":6191,"prepare.run_wall_time_us":60073,"replication_time_us":56708,"thread_start_us":44576,"threads_started":2}]]}
I20240426 13:22:24.062052 24846 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 17 to 18, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 18 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.064180 25024 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:22:24.067035 32357 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:24.139286 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.141471 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 20 to 25. New cstate: current_term: 25 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.168197 24852 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.169885 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 23 to 24, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.172941 24852 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.174408 24852 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 22 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 23 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.175555 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.203295 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 9 to 11, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 11 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.221661 24852 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 14 to 15, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 15 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:24.241813 24846 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 292) took 24292 ms (client timeout 15000 ms). Trace:
W20240426 13:22:24.242300 24846 rpcz_store.cc:269] 0426 13:21:59.949136 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:59.983399 (+ 34263us) service_pool.cc:225] Handling call
0426 13:22:24.241728 (+24258329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:24.097411 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:24.097910 (+   499us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:24.097950 (+    40us) write_op.cc:427] Acquired schema lock
0426 13:22:24.097975 (+    25us) tablet.cc:660] Decoding operations
0426 13:22:24.098480 (+   505us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:24.098538 (+    58us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:24.098568 (+    30us) tablet.cc:683] Acquiring locks for 3 operations
0426 13:22:24.098849 (+   281us) tablet.cc:699] Row locks acquired
0426 13:22:24.098873 (+    24us) write_op.cc:260] PREPARE: finished
0426 13:22:24.099006 (+   133us) write_op.cc:270] Start()
0426 13:22:24.099103 (+    97us) write_op.cc:275] Timestamp: P: 1714137744098955 usec, L: 0
0426 13:22:24.099122 (+    19us) op_driver.cc:331] REPLICATION: starting
0426 13:22:24.099658 (+   536us) log.cc:842] Serialized 1299 byte log entry
0426 13:22:24.101518 (+  1860us) op_driver.cc:444] REPLICATION: finished
0426 13:22:24.101951 (+   433us) write_op.cc:300] APPLY: starting
0426 13:22:24.102026 (+    75us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:24.103632 (+  1606us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:24.103745 (+   113us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:24.107300 (+  3555us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:24.107487 (+   187us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0426 13:22:24.107536 (+    49us) write_op.cc:311] APPLY: finished
0426 13:22:24.107736 (+   200us) log.cc:842] Serialized 41 byte log entry
0426 13:22:24.107876 (+   140us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.109093 (+  1217us) write_op.cc:446] Released schema lock
0426 13:22:24.109139 (+    46us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:24.109391 (+   252us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.109420 (+    29us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":221,"apply.run_cpu_time_us":4881,"apply.run_wall_time_us":7742,"cfile_cache_hit":12,"cfile_cache_hit_bytes":12546,"num_ops":3,"prepare.queue_time_us":2768,"prepare.run_cpu_time_us":2527,"prepare.run_wall_time_us":2527,"replication_time_us":2308}]]}
I20240426 13:22:24.245100 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.252286 24852 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 23 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 25 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.267843 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 16 to 19, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 19 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.291656 24852 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 19 to 26, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.323719 24852 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 21 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.383822 24852 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 16 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.648536 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 182) took 1938 ms. Trace:
I20240426 13:22:24.649076 24852 rpcz_store.cc:276] 0426 13:22:22.709889 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:22.711931 (+  2042us) service_pool.cc:225] Handling call
0426 13:22:24.648435 (+1936504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:24.478333 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:24.478878 (+   545us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:24.478922 (+    44us) write_op.cc:427] Acquired schema lock
0426 13:22:24.478940 (+    18us) tablet.cc:660] Decoding operations
0426 13:22:24.480311 (+  1371us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:24.480382 (+    71us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:24.480458 (+    76us) tablet.cc:683] Acquiring locks for 15 operations
0426 13:22:24.481661 (+  1203us) tablet.cc:699] Row locks acquired
0426 13:22:24.481679 (+    18us) write_op.cc:260] PREPARE: finished
0426 13:22:24.481836 (+   157us) write_op.cc:270] Start()
0426 13:22:24.481949 (+   113us) write_op.cc:275] Timestamp: P: 1714137744481787 usec, L: 0
0426 13:22:24.481968 (+    19us) op_driver.cc:331] REPLICATION: starting
0426 13:22:24.482622 (+   654us) log.cc:842] Serialized 5809 byte log entry
0426 13:22:24.489834 (+  7212us) op_driver.cc:444] REPLICATION: finished
0426 13:22:24.490887 (+  1053us) write_op.cc:300] APPLY: starting
0426 13:22:24.490975 (+    88us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:24.541904 (+ 50929us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b180034f2a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:22:24.544408 (+  2504us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:24.544449 (+    41us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:24.605067 (+ 60618us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:24.605319 (+   252us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0426 13:22:24.605367 (+    48us) write_op.cc:311] APPLY: finished
0426 13:22:24.605676 (+   309us) log.cc:842] Serialized 139 byte log entry
0426 13:22:24.605832 (+   156us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.605978 (+   146us) write_op.cc:446] Released schema lock
0426 13:22:24.606028 (+    50us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:24.612314 (+  6286us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.612344 (+    30us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":52241,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":837,"apply.run_cpu_time_us":18148,"apply.run_wall_time_us":121912,"cfile_cache_hit":60,"cfile_cache_hit_bytes":62730,"num_ops":15,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":4585,"prepare.run_wall_time_us":26515,"replication_time_us":7772,"spinlock_wait_cycles":67916672}]]}
I20240426 13:22:24.661084 24849 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 22, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.681385 24849 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 17 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.683816 24849 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.685703 24849 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 22 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.687026 24849 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 25 to 26, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 26 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.695876 24849 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.697510 24849 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 21 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 23 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.700577 24849 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 16 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.702713 24849 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 25 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.704556 24849 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 18 to 22, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 22 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.705902 24849 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 16 to 19, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.707453 24849 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 22, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.709105 24849 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.710630 24849 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.712499 24849 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.713714 24849 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 18 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.716019 24849 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:24.721498 24849 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 17 to 19, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.724354 24849 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 13 to 14, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 14 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:24.925815 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:24.941291 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.942880 24852 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:24.944213 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 19 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.056241 24849 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 293) took 2906 ms. Trace:
I20240426 13:22:25.058482 25099 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.056653 24849 rpcz_store.cc:276] 0426 13:22:22.150263 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:22.150536 (+   273us) service_pool.cc:225] Handling call
0426 13:22:25.056153 (+2905617us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:24.731738 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:24.732241 (+   503us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:24.732271 (+    30us) write_op.cc:427] Acquired schema lock
0426 13:22:24.732287 (+    16us) tablet.cc:660] Decoding operations
0426 13:22:24.733684 (+  1397us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:24.733741 (+    57us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:24.733768 (+    27us) tablet.cc:683] Acquiring locks for 19 operations
0426 13:22:24.735158 (+  1390us) tablet.cc:699] Row locks acquired
0426 13:22:24.735180 (+    22us) write_op.cc:260] PREPARE: finished
0426 13:22:24.735318 (+   138us) write_op.cc:270] Start()
0426 13:22:24.735421 (+   103us) write_op.cc:275] Timestamp: P: 1714137744735276 usec, L: 0
0426 13:22:24.735443 (+    22us) op_driver.cc:331] REPLICATION: starting
0426 13:22:24.736074 (+   631us) log.cc:842] Serialized 7408 byte log entry
0426 13:22:24.738955 (+  2881us) op_driver.cc:444] REPLICATION: finished
0426 13:22:24.739361 (+   406us) write_op.cc:300] APPLY: starting
0426 13:22:24.739439 (+    78us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:24.782818 (+ 43379us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:24.782904 (+    86us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:24.926104 (+143200us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:24.926373 (+   269us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0426 13:22:24.926422 (+    49us) write_op.cc:311] APPLY: finished
0426 13:22:24.926796 (+   374us) log.cc:842] Serialized 172 byte log entry
0426 13:22:24.931828 (+  5032us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.932003 (+   175us) write_op.cc:446] Released schema lock
0426 13:22:24.932078 (+    75us) write_op.cc:340] FINISH: Updating metrics
0426 13:22:24.932337 (+   259us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:24.932367 (+    30us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":92403,"reported_tablets":50,"spinlock_wait_cycles":13696,"child_traces":[["op",{"apply.queue_time_us":227,"apply.run_cpu_time_us":23456,"apply.run_wall_time_us":197185,"cfile_cache_hit":76,"cfile_cache_hit_bytes":79458,"num_ops":19,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":4600,"prepare.run_wall_time_us":4600,"replication_time_us":3409}]]}
I20240426 13:22:25.136466 25023 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:22:25.142609 25621 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:22:25.145448   500 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
W20240426 13:22:25.205310 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44705.2us
I20240426 13:22:25.168133 32700 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20240426 13:22:25.218250 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:25.218853 25098 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:25.223588 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:25.224224 25098 raft_consensus.cc:595] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:25.228657 32700 raft_consensus.cc:979] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:25.229151 32700 raft_consensus.cc:1067] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:25.231343 25024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:25.231913 25024 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:25.231993 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:25.232342 25024 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.232650 25098 raft_consensus.cc:595] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:25.254518 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:25.255668 25098 raft_consensus.cc:595] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:25.256644 25098 raft_consensus.cc:603] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240426 13:22:25.260634 24850 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 276) took 52785 ms (client timeout 15000 ms). Trace:
W20240426 13:22:25.261035 24850 rpcz_store.cc:269] 0426 13:21:32.475059 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:32.475977 (+   918us) service_pool.cc:225] Handling call
0426 13:22:25.260396 (+52784419us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25914,"reported_tablets":50}
I20240426 13:22:25.265532 25024 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.271435 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:25.271545 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:25.272092 25098 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:25.272270 25624 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.272477 25098 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:25.273285 25098 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.274403 25098 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.278285 25024 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:25.284289   522 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Committing config change with OpId 19.14: config changed from index 5 to 14, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.287225   301 raft_consensus.cc:979] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:25.287880   301 raft_consensus.cc:1067] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:25.289521 25621 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Committing config change with OpId 19.14: config changed from index 5 to 14, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.292891 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:25.293430 24956 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:25.293740 24956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:25.294792 25624 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:25.298462 25098 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:25.299784 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:25.300660 32225 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:25.301079 32225 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:25.301824 32225 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.323251 24956 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.339761 24956 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:25.340420 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:25.341460 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11fad756437243d2959f5998d6bf8309 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:25.341737 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:25.342305 25102 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:25.342612 25102 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:25.342823 24851 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 282) took 53082 ms (client timeout 15000 ms). Trace:
W20240426 13:22:25.343291 24851 rpcz_store.cc:269] 0426 13:21:32.260314 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:32.265619 (+  5305us) service_pool.cc:225] Handling call
0426 13:22:25.342646 (+53077027us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":170154,"reported_tablets":50}
I20240426 13:22:25.343380 25102 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.345000 25102 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:25.341101 25099 raft_consensus.cc:595] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:25.346555 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:25.348591 25022 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:25.353770 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:22:25.355603 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:25.355119 24849 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 24 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 27 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:25.358386 24849 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 16 to 18, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.359722 24849 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 25 to 27, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 27 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.360678 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:25.361055   520 raft_consensus.cc:979] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:25.361603   520 raft_consensus.cc:1067] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:25.363916 25023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:25.364429 25023 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:25.364761 25023 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Advancing to term 29
I20240426 13:22:25.369069 25024 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Committing config change with OpId 19.14: config changed from index 5 to 14, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.372097 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 5 to 14, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.419843 25023 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.422024 25023 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 29 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:25.422946 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:25.423525 25099 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 LEADER]: Stepping down as leader of term 28
I20240426 13:22:25.423768 25099 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:25.423882 25022 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:25.424562 25099 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.425130 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:25.396718 24849 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 24 to 26, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.426041   525 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:25.426407 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:25.426568   525 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:25.426964 25621 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Advancing to term 29
I20240426 13:22:25.427649   525 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.436554 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:25.445832 25621 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 29.
I20240426 13:22:25.425838 25099 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Advancing to term 29
I20240426 13:22:25.456880 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 29.
I20240426 13:22:25.457892 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:25.471546 32225 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Leader election won for term 29
I20240426 13:22:25.426443 24849 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 21 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.479977 24849 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 22 to 23, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.484876 24849 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 17 to 22, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:25.495827 24849 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:25.499409 24849 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:25.501134 32225 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:25.501092 24849 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:25.502069 32225 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:25.509349 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fad756437243d2959f5998d6bf8309 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:37292
I20240426 13:22:25.556406   534 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:25.558341   534 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:25.559793   534 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:25.563297   534 ts_tablet_manager.cc:1875] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:25.636899   301 raft_consensus.cc:979] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:25.637506   522 raft_consensus.cc:1067] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:25.639240 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:25.639847 24951 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:25.640237 24951 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.658583 24951 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.676549 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:25.677160 25022 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.685592 25022 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:25.696254 24951 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:25.697211 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:25.697460 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:25.697899 25099 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:25.698324 25099 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:25.698462   525 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:25.700556 25099 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.699394   525 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:25.702011 25099 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:25.702239   525 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:25.708722 25099 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:25.711225   534 ts_tablet_manager.cc:1888] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 19.13
I20240426 13:22:25.711699   534 log.cc:1196] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/11fad756437243d2959f5998d6bf8309
I20240426 13:22:25.713534 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 11fad756437243d2959f5998d6bf8309 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:25.744110 25099 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20240426 13:22:25.746160   532 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20240426 13:22:25.801569 24847 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 283) took 32717 ms (client timeout 15000 ms). Trace:
W20240426 13:22:25.802069 24847 rpcz_store.cc:269] 0426 13:21:53.084358 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:53.084684 (+   326us) service_pool.cc:225] Handling call
0426 13:22:25.801457 (+32716773us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54023,"reported_tablets":50}
I20240426 13:22:25.803717 24849 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 24 to 25, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 25 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:25.805747 25621 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20240426 13:22:25.817409   518 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20240426 13:22:25.938582 25022 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:25.939940   525 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:25.987444 25099 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:25.989063   517 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:25.994724 25098 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20240426 13:22:25.996511   532 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240426 13:22:26.017665 25621 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20240426 13:22:26.020068   532 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20240426 13:22:26.060990 24845 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 284) took 10825 ms. Trace:
I20240426 13:22:26.061560 24845 rpcz_store.cc:276] 0426 13:22:15.235780 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:15.277232 (+ 41452us) service_pool.cc:225] Handling call
0426 13:22:26.060887 (+10783655us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1267687,"reported_tablets":50}
I20240426 13:22:26.102666 25099 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:26.104563   525 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:26.141963 25022 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:26.143999   517 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:26.161607 24845 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 28 to 29, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 29 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:26.216816 24854 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 291) took 53504 ms (client timeout 15000 ms). Trace:
W20240426 13:22:26.217274 24854 rpcz_store.cc:269] 0426 13:21:32.712459 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:21:32.713313 (+   854us) service_pool.cc:225] Handling call
0426 13:22:26.216629 (+53503316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":129458,"reported_tablets":50}
I20240426 13:22:26.223670 24849 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 24, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:26.225340 24849 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 24 to 25, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 25 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:22:26.316874 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fad756437243d2959f5998d6bf8309 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 14)) from {username='slave'} at 127.0.0.1:37292
I20240426 13:22:26.327905 24846 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 279) took 2036 ms. Trace:
I20240426 13:22:26.330757 24846 rpcz_store.cc:276] 0426 13:22:24.291553 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:24.291767 (+   214us) service_pool.cc:225] Handling call
0426 13:22:26.327838 (+2036071us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70998,"reported_tablets":45}
I20240426 13:22:26.339092 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 11fad756437243d2959f5998d6bf8309 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:26.454767 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:26.615346 25099 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:26.617902 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:26.618459 25098 raft_consensus.cc:595] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:26.621178 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:26.621691 25098 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:26.628799 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:26.629330 25098 raft_consensus.cc:595] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:26.632782 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:26.639933 25098 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:26.652339 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:26.653009 25098 raft_consensus.cc:595] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:26.655128 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551154b9c3494e4da288affc4b88c361. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:26.655896 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:22:26.657601 25022 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:22:26.658433 24951 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20240426 13:22:26.659438 32700 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:26.660403   522 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:26.671401 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:26.672631 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:26.683635 32700 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Committing config change with OpId 18.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:26.695024 25022 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Committing config change with OpId 18.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:26.721527 32700 raft_consensus.cc:979] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.725909 24847 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 18 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:26.754963 32700 raft_consensus.cc:1067] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.757881 25024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:26.746140 24948 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Committing config change with OpId 18.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:26.779424 25024 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:26.779886 25024 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:26.796952 25024 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:26.814895   522 raft_consensus.cc:979] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.815485   522 raft_consensus.cc:1067] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.816983 25023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:26.817556 25023 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:26.817878 25023 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:26.820704 32700 raft_consensus.cc:979] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.821996 32700 raft_consensus.cc:1067] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:26.824265 25022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:26.824781 25022 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:26.825110 25022 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:26.825254 25023 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.826988 25023 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:26.828233 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:26.829099 25099 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:26.829380 25099 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:26.832485 25022 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.831981 25099 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.833408 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:26.834709 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:26.837250 24948 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:26.834256 24951 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:26.852957 25024 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:26.854786 25099 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:26.859673 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:26.861280 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:26.863287 25098 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Stepping down as leader of term 18
I20240426 13:22:26.863376 24947 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:26.863819 25098 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:26.865010 25022 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:26.857450 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:26.863302 24951 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:26.867930 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:26.864641 25098 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:26.869681 25098 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:26.876833 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:26.877437 25102 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:26.877435 25098 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 19.
I20240426 13:22:26.877985 25102 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:26.878922 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:26.881356 25102 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.886029 25102 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:26.891083 24947 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:26.899544 25099 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:26.911077 25102 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:26.891716   545 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:26.927430 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:26.895027   518 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:26.933140   518 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:26.935379   545 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:26.931950   532 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:26.953243   545 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.934370   518 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:26.947264   532 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:26.970336   532 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:26.970814 24951 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:27.008930   301 raft_consensus.cc:979] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:27.009433   301 raft_consensus.cc:1067] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:27.013584   556 raft_consensus.cc:979] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:27.016719   522 raft_consensus.cc:1067] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
W20240426 13:22:27.024112 24920 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dacba3e2733347ecb1304d307310c944. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:27.028432 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:27.028956 24948 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:27.029194 24948 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:27.034705 25097 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20240426 13:22:27.036958 25023 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20240426 13:22:27.037371 24948 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:27.040398 24948 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:27.065310 24846 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 18 to 19, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 19 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:27.068434 24846 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 22 to 23, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:27.070199 24846 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 20 to 21, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:27.092450 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59002.9us
W20240426 13:22:27.092681 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 85539.7us
I20240426 13:22:27.094303 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:27.095219 25097 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:27.095525 25097 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:27.095971 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:27.096264 25022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:27.096606 25023 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:27.096896 25022 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:27.097585 25022 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:27.096843 25097 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:27.102931 25097 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:27.115007   575 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:27.117130 25023 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:27.117996 25097 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 24.
I20240426 13:22:27.119858 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:27.120762 32735 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:27.119397   575 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20240426 13:22:27.121217 32735 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:27.122812 32735 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:27.110502 25022 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:27.124982 25022 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:27.125540 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:27.134387 25102 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:27.134722 25102 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:27.135448 25102 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:27.136615 25102 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:27.133083 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:27.144312 25624 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:27.150651 25624 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:27.143738 25102 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 20.
I20240426 13:22:27.151727 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:27.153824   532 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:27.154516   532 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:27.155563   532 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:27.160564   517 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Committing config change with OpId 22.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.162391 25098 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Committing config change with OpId 22.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.181286 24846 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 23 to 24, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:27.185153 24948 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:27.193917 24854 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 19 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 20 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:27.196336 25022 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:27.232679 24920 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ece98ef0f07f4184bedf2cb96b628f20. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:27.237824 24854 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:27.244361 25102 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20240426 13:22:27.249231   525 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:27.264165 25023 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20240426 13:22:27.268452 32735 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:27.272128   575 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Committing config change with OpId 26.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.281023 25102 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Committing config change with OpId 26.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.288479 25023 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Committing config change with OpId 26.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.301898 24846 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.315064 24948 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:22:27.336860 24920 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f04bd8996e6349879e25fca4e52dbda9. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:27.336961 25023 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:22:27.337262 25098 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20240426 13:22:27.338951   575 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240426 13:22:27.340252   555 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240426 13:22:27.357648   575 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.369037 24948 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:27.370810   532 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:27.365837 25023 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.423614 24947 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:27.436540   532 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:27.440429 25102 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.461784 24847 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 11, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:27.472676 25097 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:27.474483   532 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:22:27.477627 24995 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551154b9c3494e4da288affc4b88c361. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:27.491166 24956 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20240426 13:22:27.499953 25098 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20240426 13:22:27.519845   568 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20240426 13:22:27.514022   545 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20240426 13:22:27.597121 25022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:27.597975 25022 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Received request to transfer leadership
I20240426 13:22:27.588027 25098 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20240426 13:22:27.625145   591 ts_tablet_manager.cc:902] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:27.625811 25022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:27.626322 25022 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Received request to transfer leadership
I20240426 13:22:27.624343   568 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
W20240426 13:22:27.627846 24840 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 551154b9c3494e4da288affc4b88c361 already in progress [suppressed 1 similar messages]
I20240426 13:22:27.642735 25097 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:27.649684   555 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:27.672672 25022 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:27.676327   525 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240426 13:22:27.694918 24840 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67101us
I20240426 13:22:27.716694 25023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:27.717327 25023 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Received request to transfer leadership
I20240426 13:22:27.735023 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:22:27.738713 24901 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 551154b9c3494e4da288affc4b88c361 already in progress
W20240426 13:22:27.739121 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 551154b9c3494e4da288affc4b88c361 already in progress
I20240426 13:22:27.816906 25621 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20240426 13:22:27.819316   564 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:27.824534   599 ts_tablet_manager.cc:902] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:27.848788 25102 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20240426 13:22:27.919046   606 ts_tablet_manager.cc:902] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:27.923507   591 tablet_copy_client.cc:308] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:22:27.941289 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:27.942003 25102 raft_consensus.cc:595] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:27.935546   565 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20240426 13:22:27.955554 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet dacba3e2733347ecb1304d307310c944 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:27.956218 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet dacba3e2733347ecb1304d307310c944 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-dacba3e2733347ecb1304d307310c944
I20240426 13:22:27.970021 24966 tablet_copy_source_session.cc:205] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:27.982354   565 raft_consensus.cc:979] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:27.988425   565 raft_consensus.cc:1067] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:27.990098 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:22:27.990664 24947 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:27.991019 24947 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:27.992039   591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:27.998292 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:27.999048 25097 raft_consensus.cc:595] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:28.019068 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:28.019800 25097 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:28.030308 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:28.030994 25097 raft_consensus.cc:595] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:28.032162 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:22:28.033866 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:28.034502 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:28.048812 24947 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:28.112126 24947 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:28.120080 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:28.122324 25097 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:28.123108 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:28.123765 25024 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:28.124061 25024 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:28.124416   301 raft_consensus.cc:979] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:28.124781 25024 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:28.147857 25024 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:28.159036 25097 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:28.195225 25024 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:28.199646 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:28.180665   560 raft_consensus.cc:1067] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:28.217850 32735 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
I20240426 13:22:28.218371 32735 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:28.219360 32735 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:28.219477   591 tablet_copy_client.cc:783] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:28.231206   614 ts_tablet_manager.cc:902] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:28.242623   591 tablet_copy_client.cc:647] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:28.280537   606 tablet_copy_client.cc:308] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:22:28.281684 25024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:28.285878 25024 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:28.286260 25024 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:28.292728 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93133.1us
I20240426 13:22:28.300786 24854 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 19 to 20, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:28.325191 25024 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:28.329210 25024 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:28.330255 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:28.331048 25620 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:28.347689 25620 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:28.350281 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:28.360486   619 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:28.361200   619 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:28.360596 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet f04bd8996e6349879e25fca4e52dbda9 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:28.365209 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:28.365810 25097 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:28.366077 25097 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:28.366827 25097 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:28.370123 25097 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:28.396903   599 tablet_copy_client.cc:308] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:22:28.397907 25097 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:28.404850 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet f04bd8996e6349879e25fca4e52dbda9 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-f04bd8996e6349879e25fca4e52dbda9
I20240426 13:22:28.408545   619 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:28.520483   614 tablet_copy_client.cc:308] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:22:28.619714 24965 tablet_copy_source_session.cc:205] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:28.671931   624 raft_consensus.cc:979] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:28.672705   624 raft_consensus.cc:1067] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:28.703095 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
W20240426 13:22:28.707494 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51864.6us
I20240426 13:22:28.703912 24949 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:28.708251 24949 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Advancing to term 18
W20240426 13:22:28.712291 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 51022.9us
I20240426 13:22:28.713260   301 raft_consensus.cc:979] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:28.713760   617 raft_consensus.cc:1067] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:28.717700 24949 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:28.720718 24949 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:28.721877 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet ece98ef0f07f4184bedf2cb96b628f20 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:28.722471 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet ece98ef0f07f4184bedf2cb96b628f20 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-ece98ef0f07f4184bedf2cb96b628f20
I20240426 13:22:28.725279 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:28.725893 24956 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:28.726255 24956 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:28.732977 24956 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:28.735771 24956 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:28.753930 25099 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20240426 13:22:28.765807 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:28.766844 25099 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Stepping down as leader of term 17
I20240426 13:22:28.767323 25099 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:22:28.767349 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 356408us
I20240426 13:22:28.788875 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:28.789678 25102 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:28.790025 25102 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:28.847311 25099 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:28.859707 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62202us
I20240426 13:22:28.860080 25102 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:28.875034 25102 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:28.885284 24965 tablet_copy_source_session.cc:205] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:28.922253 25102 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:28.937765 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:28.938418 25621 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:28.956928 25621 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
W20240426 13:22:28.960533 24920 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551154b9c3494e4da288affc4b88c361. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:28.966564 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 
I20240426 13:22:28.983330   621 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Leader election won for term 18
I20240426 13:22:28.985025 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 551154b9c3494e4da288affc4b88c361 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:22:28.985671 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 551154b9c3494e4da288affc4b88c361 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-551154b9c3494e4da288affc4b88c361
I20240426 13:22:28.993760 25099 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 17 FOLLOWER]: Advancing to term 18
I20240426 13:22:29.001714 25099 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 18.
I20240426 13:22:29.019516 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:29.014557   525 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.007s
I20240426 13:22:29.003505   621 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:29.030478   621 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:29.032052   591 tablet_copy_client.cc:523] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:29.027824   574 raft_consensus.cc:979] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:29.029417 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:29.035341 25031 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:29.038496 32735 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:29.034390 25041 tablet_copy_source_session.cc:205] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:29.040556 32735 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:29.043570   606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:29.044775   650 raft_consensus.cc:1067] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:29.046950 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:29.047562 24947 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:29.048094 24947 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:29.048728 25031 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:29.053493 32735 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.068289   599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:22:29.074743 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54944.7us
I20240426 13:22:29.077502 24947 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.079329 24947 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:29.086423 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:29.087095 25030 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:29.088491 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:29.089130 25103 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:29.089417 25103 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:29.090196 25103 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.091459 25103 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:29.094697 25024 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20240426 13:22:29.099948   614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:29.103696 25030 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:29.117450 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:29.130174 25102 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:22:29.130458 25024 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 22. Current term is 23. Ops: []
I20240426 13:22:29.131348 24847 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 20 to 21, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:29.155218   591 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:29.157496   599 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44512 (request call id 1683) took 1535 ms. Trace:
I20240426 13:22:29.158126   599 rpcz_store.cc:276] 0426 13:22:27.622013 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:27.622150 (+   137us) service_pool.cc:225] Handling call
0426 13:22:27.824921 (+202771us) ts_tablet_manager.cc:903] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
0426 13:22:29.066674 (+1241753us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:29.127498 (+ 60824us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:29.127753 (+   255us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:29.156566 (+ 28813us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:29.156911 (+   345us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:29.157318 (+   407us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:29.157393 (+    75us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":43397,"tablet-copy.queue_time_us":201860,"thread_start_us":613260,"threads_started":5}
I20240426 13:22:29.160272   599 tablet_copy_client.cc:783] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:29.164883   599 tablet_copy_client.cc:647] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:29.164564   619 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:29.170421   599 tablet_copy_client.cc:523] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:29.216495 25103 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:29.225395 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:29.243345   646 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:29.243938   646 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:29.244853   646 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.270694 24854 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 295) took 1016 ms. Trace:
I20240426 13:22:29.271716   599 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:29.273101 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:29.273734 25099 raft_consensus.cc:595] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:29.271147 24854 rpcz_store.cc:276] 0426 13:22:28.254258 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:28.254408 (+   150us) service_pool.cc:225] Handling call
0426 13:22:29.270596 (+1016188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:22:28.647177 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:22:28.647694 (+   517us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:22:28.647729 (+    35us) write_op.cc:427] Acquired schema lock
0426 13:22:28.647746 (+    17us) tablet.cc:660] Decoding operations
0426 13:22:28.648079 (+   333us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:22:28.648153 (+    74us) write_op.cc:633] Partition lock acquired for write op
0426 13:22:28.648181 (+    28us) tablet.cc:683] Acquiring locks for 1 operations
0426 13:22:28.648350 (+   169us) tablet.cc:699] Row locks acquired
0426 13:22:28.648368 (+    18us) write_op.cc:260] PREPARE: finished
0426 13:22:28.648502 (+   134us) write_op.cc:270] Start()
0426 13:22:28.648598 (+    96us) write_op.cc:275] Timestamp: P: 1714137748648455 usec, L: 0
0426 13:22:28.648618 (+    20us) op_driver.cc:331] REPLICATION: starting
0426 13:22:28.649220 (+   602us) log.cc:842] Serialized 591 byte log entry
0426 13:22:29.013883 (+364663us) op_driver.cc:444] REPLICATION: finished
0426 13:22:29.174402 (+160519us) write_op.cc:300] APPLY: starting
0426 13:22:29.174493 (+    91us) tablet.cc:1365] starting BulkCheckPresence
0426 13:22:29.175557 (+  1064us) tablet.cc:1368] finished BulkCheckPresence
0426 13:22:29.175671 (+   114us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:22:29.176442 (+   771us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:22:29.176693 (+   251us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=2,mrs_lookups=0
0426 13:22:29.176760 (+    67us) write_op.cc:311] APPLY: finished
0426 13:22:29.177004 (+   244us) log.cc:842] Serialized 25 byte log entry
0426 13:22:29.228577 (+ 51573us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:22:29.233432 (+  4855us) write_op.cc:446] Released schema lock
0426 13:22:29.239097 (+  5665us) write_op.cc:340] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":150555,"apply.run_cpu_time_us":3761,"apply.run_wall_time_us":100556,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8309,"num_ops":1,"prepare.queue_time_us":337205,"prepare.run_cpu_time_us":4369,"prepare.run_wall_time_us":74639,"raft.queue_time_us":90369,"raft.run_cpu_time_us":24238,"raft.run_wall_time_us":179898,"replication_time_us":364269,"thread_start_us":206415,"threads_started":4,"wal-append.queue_time_us":229509}]]}
I20240426 13:22:29.289011   645 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:29.308009 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:29.308760 25099 raft_consensus.cc:595] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:29.350131   606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44512 (request call id 1686) took 1486 ms. Trace:
I20240426 13:22:29.350652   606 rpcz_store.cc:276] 0426 13:22:27.863885 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:27.863997 (+   112us) service_pool.cc:225] Handling call
0426 13:22:27.919356 (+ 55359us) ts_tablet_manager.cc:903] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
0426 13:22:29.042151 (+1122795us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:29.179578 (+137427us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:29.179828 (+   250us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:29.349394 (+169566us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:29.349709 (+   315us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:29.350014 (+   305us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:29.350067 (+    53us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":294,"tablet-copy.queue_time_us":54553,"thread_start_us":381812,"threads_started":5}
I20240426 13:22:29.352252   606 tablet_copy_client.cc:783] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:29.353127   606 tablet_copy_client.cc:647] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:29.386266   614 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44520 (request call id 1165) took 1300 ms. Trace:
I20240426 13:22:29.386744   614 rpcz_store.cc:276] 0426 13:22:28.086203 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:28.086354 (+   151us) service_pool.cc:225] Handling call
0426 13:22:28.242494 (+156140us) ts_tablet_manager.cc:903] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
0426 13:22:29.096648 (+854154us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:29.200727 (+104079us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:29.200990 (+   263us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:29.385517 (+184527us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:29.385831 (+   314us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:29.386145 (+   314us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:29.386205 (+    60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":25589,"tablet-copy.queue_time_us":113536,"thread_start_us":214374,"threads_started":5}
I20240426 13:22:29.391184   614 tablet_copy_client.cc:783] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:29.415467   614 tablet_copy_client.cc:647] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:29.427755 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:29.430505 25105 raft_consensus.cc:595] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:29.430889 25105 raft_consensus.cc:603] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:29.450232 25621 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20240426 13:22:29.470729 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:29.471354 25105 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:29.471704 25105 raft_consensus.cc:603] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:22:29.475867 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:22:29.477519 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:29.478165 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:29.487462   619 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:29.491945   614 tablet_copy_client.cc:523] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:29.500384   606 tablet_copy_client.cc:523] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:29.501863 24846 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 22 to 23, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:29.506800 24846 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 17 to 18, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:29.509299   614 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:29.508451 24846 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 22 to 23, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:29.604705 25620 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:22:29.608704 32700 raft_consensus.cc:979] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:29.609223 32700 raft_consensus.cc:1067] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:29.627928 32735 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:29.646867 25105 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20240426 13:22:29.659222   624 raft_consensus.cc:979] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:29.659718   624 raft_consensus.cc:1067] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:29.661082 25105 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:22:29.664440 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:29.665056 24947 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:29.664777   646 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:29.665414 24947 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:29.683121   606 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:29.700371 25030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51366
I20240426 13:22:29.700996 25030 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:29.701344 25030 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:29.683302 24947 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.704723 24947 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:29.708109 25030 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.713063 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:29.713821 25105 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:29.714179 25105 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:29.715010 25105 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.716840 25105 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:29.717711 25030 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:29.662824   626 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20240426 13:22:29.723883 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:29.724499 25099 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Stepping down as leader of term 19
I20240426 13:22:29.724802 25099 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:29.725098 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:29.727344 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:29.727613 25099 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.732951 25031 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20240426 13:22:29.733741   619 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:29.734334   619 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:29.735492 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:29.736423 24947 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:29.738890 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:29.739449 25024 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20240426 13:22:29.739473 25031 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:29.740630 25105 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20240426 13:22:29.748220 25099 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:29.780181 25099 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:29.804514 24947 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 21.
I20240426 13:22:29.809949 32735 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20240426 13:22:29.805495 25031 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 20.
I20240426 13:22:29.800410   645 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20240426 13:22:29.795140   619 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:29.866627   645 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:29.867765 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:29.872200   517 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader election won for term 20
W20240426 13:22:29.947340 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69281us
I20240426 13:22:29.926288   517 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:29.990433   517 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:30.022316 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44783.2us
I20240426 13:22:30.066664   591 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=2 applied=14 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:30.067622   591 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:30.088799 24854 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 19 to 20, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:30.148561   682 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20240426 13:22:30.105562   591 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.951s	user 0.045s	sys 0.023s
I20240426 13:22:30.152102 24847 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 20 to 21, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 21 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:30.201159   657 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.002s
W20240426 13:22:30.250767 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106369us
W20240426 13:22:30.270912   685 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20240426 13:22:30.322942   591 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.323992   591 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:30.324680   591 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.373148   591 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.212s	user 0.003s	sys 0.004s
I20240426 13:22:30.404350 24947 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20240426 13:22:30.402036 25099 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20240426 13:22:30.411237   565 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:30.413569   565 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:30.419855 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-dacba3e2733347ecb1304d307310c944 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:30.420334 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-dacba3e2733347ecb1304d307310c944 on tablet dacba3e2733347ecb1304d307310c944 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:30.461534 25621 raft_consensus.cc:1203] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Deduplicated request from leader. Original: 22.13->[22.14-22.14]   Dedup: 22.14->[]
I20240426 13:22:30.477387 25024 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:30.526080 25105 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20240426 13:22:30.494779   707 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20240426 13:22:30.544234   707 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240426 13:22:30.551144   599 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=1 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:30.552139   599 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
W20240426 13:22:30.581547   606 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.073s	user 0.003s	sys 0.000s
I20240426 13:22:30.584952   599 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 1.313s	user 0.038s	sys 0.029s
I20240426 13:22:30.608619   599 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.609578   599 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:30.610312   599 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.614945   614 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:30.618850   599 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.002s
I20240426 13:22:30.628428 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-ece98ef0f07f4184bedf2cb96b628f20 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:30.629082 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-ece98ef0f07f4184bedf2cb96b628f20 on tablet ece98ef0f07f4184bedf2cb96b628f20 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:30.615906   614 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:30.651762   614 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 1.143s	user 0.051s	sys 0.018s
I20240426 13:22:30.705327   614 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.708623 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:30.724157   614 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:30.726394   614 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.729131   614 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.063s	user 0.005s	sys 0.001s
I20240426 13:22:30.734582   606 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:30.735513   606 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:30.736296   606 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 1.053s	user 0.070s	sys 0.015s
I20240426 13:22:30.748072 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-551154b9c3494e4da288affc4b88c361 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:22:30.748549 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-551154b9c3494e4da288affc4b88c361 on tablet 551154b9c3494e4da288affc4b88c361 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:30.766891   606 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.768348   606 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:30.769062   606 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:30.771572   606 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.002s
I20240426 13:22:30.773236 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f04bd8996e6349879e25fca4e52dbda9 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:30.773656 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f04bd8996e6349879e25fca4e52dbda9 on tablet f04bd8996e6349879e25fca4e52dbda9 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:30.806972 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:30.807435 25621 raft_consensus.cc:1203] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEARNER]: Deduplicated request from leader. Original: 18.10->[18.11-18.11]   Dedup: 18.11->[]
I20240426 13:22:30.807582 24948 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:30.810843 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:30.811412 24948 raft_consensus.cc:595] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:30.815027 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:30.815583 24948 raft_consensus.cc:595] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:30.821027 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 13:22:30.824223 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:30.824836 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:30.833180 25620 raft_consensus.cc:1203] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 LEARNER]: Deduplicated request from leader. Original: 26.15->[26.16-26.16]   Dedup: 26.16->[]
I20240426 13:22:30.883548   707 raft_consensus.cc:1050] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:30.887815   707 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 12, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:30.930404 25620 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:30.932364   707 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:30.933449 25104 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:30.937840   694 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:30.941260 25024 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:30.944221   707 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:31.000633   694 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Committing config change with OpId 22.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.006078 25620 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Committing config change with OpId 22.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.008363 25031 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.019543   626 raft_consensus.cc:979] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:31.020102   626 raft_consensus.cc:1067] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:31.026032 24846 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.030133 25104 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Committing config change with OpId 22.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.032322 25099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58866
I20240426 13:22:31.032904 25099 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:31.033229 25099 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.052772 25099 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.054451 25099 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:31.055632 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:31.056901 25030 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.074414 25030 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:31.062072 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:31.076792 24948 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:31.077207 24948 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:31.075911 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:31.078094 24948 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.084890 24948 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.085229   654 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:31.160931   646 raft_consensus.cc:979] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:31.161473   646 raft_consensus.cc:1067] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:31.163187 25099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58866
I20240426 13:22:31.163890 25099 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:31.164279 25099 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:31.187883 25099 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:31.190580 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:31.191215 24956 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Stepping down as leader of term 20
I20240426 13:22:31.191557 24956 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:31.192353 24956 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:31.194672 24956 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:31.218147 25099 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:31.225464 24956 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:31.226601 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:31.228545 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:31.232223   654 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:31.253486   654 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.267513   723 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:31.269132   723 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:31.270190   723 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:31.292209 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:31.292892 25030 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:31.300581 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:31.318926 24948 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:31.374850 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78270.8us
W20240426 13:22:31.415760 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56656.8us
W20240426 13:22:31.419114 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103949us
I20240426 13:22:31.425899   646 raft_consensus.cc:1050] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:31.457248   646 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 9, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:31.468683 25624 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20240426 13:22:31.522224 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 204511us
I20240426 13:22:31.513203   707 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20240426 13:22:31.575776 24847 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:31.587560 24847 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:31.590517 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75282.9us
I20240426 13:22:31.610268   732 raft_consensus.cc:979] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:31.610921   626 raft_consensus.cc:1067] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:31.612299 25028 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20240426 13:22:31.616901 25028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:22:31.617444 25028 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:31.617815 25028 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.625329   703 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20240426 13:22:31.651691 25028 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.661944 25028 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:31.686944   729 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Committing config change with OpId 22.16: config changed from index 15 to 16, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.695822 25105 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:22:31.700985 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:31.701552 24949 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:31.701844 24949 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:31.703099 25029 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:22:31.684974   646 raft_consensus.cc:1050] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:31.702811 24949 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.705878 24949 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.710566 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:31.711083 25099 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:31.717648 25099 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:31.713783   727 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20240426 13:22:31.722491 25027 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.16: config changed from index 15 to 16, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.723800   707 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:31.724545 25624 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20240426 13:22:31.732148   707 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240426 13:22:31.736187 25620 raft_consensus.cc:2932] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Committing config change with OpId 22.16: config changed from index 15 to 16, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.741518   646 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 14, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:31.758152 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 168350us
I20240426 13:22:31.762244 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dacba3e2733347ecb1304d307310c944 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:31.755730 24847 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 15 to 16, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.785187 25024 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20240426 13:22:31.788146 24949 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:31.789147 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:31.790817   738 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:31.793149   738 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:31.794009   738 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:31.795466   750 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Committing config change with OpId 18.12: config changed from index 11 to 12, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.790391   735 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.012s
I20240426 13:22:31.804103 25624 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Committing config change with OpId 18.12: config changed from index 11 to 12, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.805768 24948 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20240426 13:22:31.807442   723 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240426 13:22:31.830905 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet dacba3e2733347ecb1304d307310c944 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:58852
I20240426 13:22:31.832695 25099 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Committing config change with OpId 18.12: config changed from index 11 to 12, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:31.848601 25027 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:31.867169 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:31.868969   723 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:22:31.908192 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51801.4us
I20240426 13:22:31.912181   723 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:31.920778   761 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:31.922683   761 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:31.923911   761 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:31.929953   761 ts_tablet_manager.cc:1875] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:31.956353 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:31.965647 24849 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 11 to 12, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:31.967728 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83094.3us
I20240426 13:22:32.021616 25624 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20240426 13:22:32.032351 25620 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Advancing to term 21
I20240426 13:22:32.021747 25099 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20240426 13:22:32.058555   703 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:32.080101   729 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:32.128561 24854 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:32.158208 25620 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20240426 13:22:32.185999   752 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20240426 13:22:32.246110 25027 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20240426 13:22:32.248023 25029 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Committing config change with OpId 18.12: config changed from index 11 to 12, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.282675 24950 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20240426 13:22:32.303495   761 ts_tablet_manager.cc:1888] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 22.15
I20240426 13:22:32.306463   761 log.cc:1196] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/dacba3e2733347ecb1304d307310c944
I20240426 13:22:32.310079 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet dacba3e2733347ecb1304d307310c944 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:32.321727   729 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:32.346385   765 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:22:32.390494 24947 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:32.393887   727 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.402143 25099 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20240426 13:22:32.403754   638 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:22:32.418706   735 raft_consensus.cc:1050] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:32.422178   735 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 17, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:32.424782 25622 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.451303   703 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.454704 25620 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20240426 13:22:32.462858 25620 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 21 index: 19. (index mismatch)
I20240426 13:22:32.468849 25027 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 21 index: 19. (index mismatch)
I20240426 13:22:32.470219 25023 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.473095   752 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240426 13:22:32.473995   650 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20240426 13:22:32.492374   694 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240426 13:22:32.501812 24847 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.505095 25101 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.516536   724 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Committing config change with OpId 21.19: config changed from index 15 to 19, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.526952 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f04bd8996e6349879e25fca4e52dbda9 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:32.557298 25023 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Committing config change with OpId 21.19: config changed from index 15 to 19, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.578325 24846 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 12 to 13, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 18 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.587567 25621 raft_consensus.cc:2932] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:32.636804   778 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20240426 13:22:32.643702 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet f04bd8996e6349879e25fca4e52dbda9 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:58852
I20240426 13:22:32.643450 25029 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.657559 24849 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 15 to 19, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.644973   761 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:32.661100   761 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:32.661973   761 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:32.668452 24948 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 21 index: 19. (index mismatch)
I20240426 13:22:32.671190   735 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240426 13:22:32.671051 25622 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Committing config change with OpId 21.19: config changed from index 15 to 19, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.677111 24947 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:32.676837 24948 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Committing config change with OpId 21.19: config changed from index 15 to 19, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.684552   761 ts_tablet_manager.cc:1875] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:32.684871 25101 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20240426 13:22:32.686235   727 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240426 13:22:32.706158 25620 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20240426 13:22:32.710927   727 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Committing config change with OpId 26.18: config changed from index 17 to 18, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.719187 25101 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Committing config change with OpId 26.18: config changed from index 17 to 18, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.724088   703 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20240426 13:22:32.730804 25621 raft_consensus.cc:2932] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Committing config change with OpId 26.18: config changed from index 17 to 18, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:32.757040 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ece98ef0f07f4184bedf2cb96b628f20 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:32.777019 25023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:32.777678 25023 raft_consensus.cc:595] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:32.784206 24846 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 17 to 18, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 26 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:32.789265 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:32.805344   761 ts_tablet_manager.cc:1888] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 18.12
I20240426 13:22:32.805999   761 log.cc:1196] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/f04bd8996e6349879e25fca4e52dbda9
I20240426 13:22:32.807474 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:32.811136 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:32.807830 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet f04bd8996e6349879e25fca4e52dbda9 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:32.817812 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:32.861524   780 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:32.915995   780 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:32.916958   780 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:32.924162   780 ts_tablet_manager.cc:1875] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240426 13:22:32.936308 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71756.6us
I20240426 13:22:32.957955   780 ts_tablet_manager.cc:1888] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 26.17
I20240426 13:22:32.958431   780 log.cc:1196] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/ece98ef0f07f4184bedf2cb96b628f20
I20240426 13:22:32.968475 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet ece98ef0f07f4184bedf2cb96b628f20 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:33.055459   737 raft_consensus.cc:979] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:33.056088   737 raft_consensus.cc:1067] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:33.058100 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb99666c521d49ada45165233068daf3"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:22:33.058746 24947 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:33.059125 24947 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:33.083487 24947 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:33.087287 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:33.087960 25029 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Stepping down as leader of term 24
I20240426 13:22:33.090842 25029 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:33.091548 25029 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:33.092954 25029 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:33.110591 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:33.111222 25101 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:33.111625 24947 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:33.113054 25029 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:33.114272 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:33.115342   703 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:33.115849   703 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:33.116714   703 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:33.118304 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 25.
I20240426 13:22:33.140492 24846 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 24 to 25, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 25 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:33.212563 25105 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 21.19, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:33.219586 25622 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 21 index: 20. (index mismatch)
I20240426 13:22:33.219586 24947 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 21 index: 19. Preceding OpId from leader: term: 21 index: 20. (index mismatch)
I20240426 13:22:33.220991   650 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240426 13:22:33.227963   752 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240426 13:22:33.230345   735 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Committing config change with OpId 21.20: config changed from index 19 to 20, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:33.254281 25622 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Committing config change with OpId 21.20: config changed from index 19 to 20, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:33.257539 24948 raft_consensus.cc:2932] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Committing config change with OpId 21.20: config changed from index 19 to 20, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:33.278494 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551154b9c3494e4da288affc4b88c361 with cas_config_opid_index 19: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:33.291285 24846 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 19 to 20, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:33.333269 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 551154b9c3494e4da288affc4b88c361 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 20) from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:33.334326   780 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:33.335433   780 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:33.336592   780 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:33.347666   780 ts_tablet_manager.cc:1875] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:33.385310   780 ts_tablet_manager.cc:1888] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 21.19
I20240426 13:22:33.385728   780 log.cc:1196] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/551154b9c3494e4da288affc4b88c361
I20240426 13:22:33.387684 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 551154b9c3494e4da288affc4b88c361 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:33.594316 25105 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20240426 13:22:33.597296   626 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:33.632830 25023 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20240426 13:22:33.635351   626 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:34.020345 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:34.067175 25023 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:22:34.109578 24995 consensus_peers.cc:485] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f4cb6239dd94f7693fdaacdc8ae8a8f. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:34.111851 25105 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:34.115686   786 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:34.118278 24948 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20240426 13:22:34.116137 24947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59632e528afa44ca98a626355e1e4660"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:34.120507 24947 raft_consensus.cc:595] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:34.122795   737 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:34.134642 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:34.136680 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:34.137305 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:34.165757   786 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Committing config change with OpId 24.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.170001 25099 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Committing config change with OpId 24.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.185318 24947 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:34.203409 24846 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.218451 24950 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:34.239848 24920 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aca59ad77ad642c48a7e7eed1635089d. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:34.298640 25023 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:34.299810 25105 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:34.302277   626 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:34.304412   626 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:34.337229   626 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Committing config change with OpId 20.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.347828 25031 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Committing config change with OpId 20.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.405081 25099 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Committing config change with OpId 20.14: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.411789 25101 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:34.430730 24849 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 14, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:34.457944 25069 consensus_peers.cc:485] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79bb183792fd4ae08dd4564ec82efc36. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:34.485080 25031 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:34.487617 24947 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:34.491729   626 raft_consensus.cc:979] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:34.491876   752 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:34.492488   626 raft_consensus.cc:1067] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:34.497298 25023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59632e528afa44ca98a626355e1e4660"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:22:34.497792 25023 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:34.501317 25023 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:34.499691   796 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20240426 13:22:34.541128 25023 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:34.544742 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:34.545531 24950 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Stepping down as leader of term 26
I20240426 13:22:34.545897 24950 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:34.546900 24950 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:34.548147 24950 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:34.546934 25023 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:34.548812 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:34.549460 25621 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:34.563380 24950 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:34.565418 25621 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:34.573158 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:34.595085   792 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:34.592685   809 ts_tablet_manager.cc:902] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:34.597558   792 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:34.598778   796 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.601725 25031 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.598932   792 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:34.646862 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 73869.5us
W20240426 13:22:34.654937 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 64175.4us
I20240426 13:22:34.665947 24849 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:34.708935 24950 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:22:34.725185 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59639us
W20240426 13:22:34.745173 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47549.3us
I20240426 13:22:34.759702 24854 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 26 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 27 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:34.768011 24949 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:34.819226 24920 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 242110e36ca34176b9d522087d440de6. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:34.830351 25104 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20240426 13:22:34.866066   794 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:34.877213   809 tablet_copy_client.cc:308] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:22:34.877647 25031 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20240426 13:22:34.881354   826 ts_tablet_manager.cc:902] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:34.881072   822 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:34.881286 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:22:34.887365 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-5f4cb6239dd94f7693fdaacdc8ae8a8f
I20240426 13:22:34.895462   822 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Committing config change with OpId 27.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.896711 25099 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Committing config change with OpId 27.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.898289 25041 tablet_copy_source_session.cc:205] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:34.901715   809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:34.908634   828 ts_tablet_manager.cc:902] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:34.921427 24854 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 27 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:34.918535 25031 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Committing config change with OpId 27.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:35.006675   809 tablet_copy_client.cc:783] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:35.010164   809 tablet_copy_client.cc:647] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:35.022890   809 tablet_copy_client.cc:523] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:35.046625 25621 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20240426 13:22:35.048799   792 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20240426 13:22:35.063674   826 tablet_copy_client.cc:308] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:22:35.069419   809 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:35.072245 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet aca59ad77ad642c48a7e7eed1635089d from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:35.072865 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet aca59ad77ad642c48a7e7eed1635089d from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-aca59ad77ad642c48a7e7eed1635089d
I20240426 13:22:35.114277 24965 tablet_copy_source_session.cc:205] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:35.129626   826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:35.185688 24950 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20240426 13:22:35.187829   786 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20240426 13:22:35.198303 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:35.225450   828 tablet_copy_client.cc:308] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:22:35.249137 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 79bb183792fd4ae08dd4564ec82efc36 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:22:35.249732 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 79bb183792fd4ae08dd4564ec82efc36 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-79bb183792fd4ae08dd4564ec82efc36
I20240426 13:22:35.342824 25114 tablet_copy_source_session.cc:205] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:35.527981 25023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:35.529314 25023 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:35.543254   846 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.210s	user 0.000s	sys 0.009s
I20240426 13:22:35.545351   847 ts_tablet_manager.cc:902] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:22:35.597954 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:35.599221 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:35.599812 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:22:35.617739 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 176895us
I20240426 13:22:35.617228   826 tablet_copy_client.cc:783] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:35.664865   826 tablet_copy_client.cc:647] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:35.668330   807 raft_consensus.cc:979] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:35.668887   807 raft_consensus.cc:1067] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:35.685556   828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:35.690687 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:22:35.691237 24948 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:35.691592 24948 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
W20240426 13:22:35.690304 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 346343us
W20240426 13:22:35.694128 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 161079us
I20240426 13:22:35.736909 24948 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:35.749498   826 tablet_copy_client.cc:523] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:35.755122 24948 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:35.796167   809 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20240426 13:22:35.821182   826 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
W20240426 13:22:35.825125 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91346.9us
I20240426 13:22:35.829838 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:35.830494 25030 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:35.830767 25030 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:22:35.831837 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100691us
I20240426 13:22:35.831460 25030 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:35.844533 25030 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:35.899412   809 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=1 applied=14 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:35.909749 25030 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:35.913375   809 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:35.914214   809 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.845s	user 0.057s	sys 0.013s
I20240426 13:22:35.924782 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:35.925781   866 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:35.926198 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:35.926256   866 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:35.926812 25099 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:35.928030   866 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:35.926256   809 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:35.932338   809 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:35.934019   809 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:35.936161 25099 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 23.
I20240426 13:22:35.947175   847 tablet_copy_client.cc:308] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:22:35.953591   809 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.004s
I20240426 13:22:35.961966 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-5f4cb6239dd94f7693fdaacdc8ae8a8f received from {username='slave'} at 127.0.0.1:47174
I20240426 13:22:35.962718 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-5f4cb6239dd94f7693fdaacdc8ae8a8f on tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:35.966436 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 242110e36ca34176b9d522087d440de6 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:35.967265 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 242110e36ca34176b9d522087d440de6 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-242110e36ca34176b9d522087d440de6
I20240426 13:22:35.967495   828 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44496 (request call id 1501) took 1091 ms. Trace:
I20240426 13:22:35.968710   828 rpcz_store.cc:276] 0426 13:22:34.876363 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:34.876510 (+   147us) service_pool.cc:225] Handling call
0426 13:22:34.909043 (+ 32533us) ts_tablet_manager.cc:903] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
0426 13:22:35.683815 (+774772us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:35.779027 (+ 95212us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:35.779282 (+   255us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:35.966274 (+186992us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:35.967046 (+   772us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:35.967352 (+   306us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:35.967416 (+    64us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1293,"tablet-copy.queue_time_us":31608,"thread_start_us":285741,"threads_started":5}
I20240426 13:22:35.994414 24849 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 22 to 23, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 23 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:35.998627   828 tablet_copy_client.cc:783] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:35.999303   828 tablet_copy_client.cc:647] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:36.003716 24965 tablet_copy_source_session.cc:205] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:36.004954   828 tablet_copy_client.cc:523] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:36.017582   847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:36.033740   828 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:36.076385   847 tablet_copy_client.cc:783] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:36.079589   847 tablet_copy_client.cc:647] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:36.106985   847 tablet_copy_client.cc:523] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:36.203787   826 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:36.213526   847 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:36.224691   826 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:36.251564   826 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.431s	user 0.062s	sys 0.001s
I20240426 13:22:36.331064 25099 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:36.336242   822 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:36.335440   826 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.342818   826 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:36.353761 25029 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20240426 13:22:36.360360   626 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:36.347136   826 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.385864   826 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.134s	user 0.008s	sys 0.000s
I20240426 13:22:36.391171 25624 raft_consensus.cc:1203] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Deduplicated request from leader. Original: 24.13->[24.14-24.14]   Dedup: 24.14->[]
I20240426 13:22:36.400367 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-aca59ad77ad642c48a7e7eed1635089d received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:36.400800 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-aca59ad77ad642c48a7e7eed1635089d on tablet aca59ad77ad642c48a7e7eed1635089d with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:36.574394   828 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:36.604827   828 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:36.605707   828 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.572s	user 0.057s	sys 0.033s
I20240426 13:22:36.608562   828 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.617988   828 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:36.624269   828 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.621022   847 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=3 applied=15 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:36.632953   847 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:36.633745   847 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.420s	user 0.049s	sys 0.048s
I20240426 13:22:36.634038   828 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.002s
I20240426 13:22:36.638494   847 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.639714   847 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:36.640381   847 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:36.640224 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-79bb183792fd4ae08dd4564ec82efc36 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:22:36.643213 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-79bb183792fd4ae08dd4564ec82efc36 on tablet 79bb183792fd4ae08dd4564ec82efc36 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:36.687593 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:36.705696   847 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.072s	user 0.008s	sys 0.000s
I20240426 13:22:36.707620 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-242110e36ca34176b9d522087d440de6 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:36.708020 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-242110e36ca34176b9d522087d440de6 on tablet 242110e36ca34176b9d522087d440de6 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:36.755508 25624 raft_consensus.cc:1203] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Deduplicated request from leader. Original: 20.13->[20.14-20.14]   Dedup: 20.14->[]
I20240426 13:22:36.780361 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:36.781112 24956 raft_consensus.cc:595] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:36.785822 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:36.787887 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:36.788632 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:36.791910 25624 raft_consensus.cc:1203] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEARNER]: Deduplicated request from leader. Original: 27.14->[27.15-27.15]   Dedup: 27.15->[]
I20240426 13:22:37.032363   626 raft_consensus.cc:979] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:37.032855   626 raft_consensus.cc:1067] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:37.044512   792 raft_consensus.cc:1050] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:37.047441   792 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 12, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.072178 25029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:22:37.072788 25029 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:37.073160 25029 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:37.079875 25029 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.081640 25029 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:37.087631 25099 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:37.092377 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:37.092445 25621 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:37.095263 25624 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:37.096153   792 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:37.101523 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:37.102201 24947 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Stepping down as leader of term 23
I20240426 13:22:37.102496 24947 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:37.103502 25624 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:37.103257 24947 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.112675 24947 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:37.113830   786 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:37.116199 24948 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:37.121472 24947 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:37.161600 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:37.168439   792 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20240426 13:22:37.173527 25621 raft_consensus.cc:1203] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Deduplicated request from leader. Original: 24.14->[24.15-24.15]   Dedup: 24.15->[]
I20240426 13:22:37.168217   843 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:37.175791   843 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:37.176712   843 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.207528   856 raft_consensus.cc:1050] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:37.210178   856 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.224014 24849 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:37.228466   890 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Committing config change with OpId 24.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.235752 25621 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.261463 24854 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.264542 25621 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20240426 13:22:37.271775 25104 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20240426 13:22:37.273757   866 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:37.280619   866 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:37.280702 24948 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.304618 25099 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.313911 25030 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20240426 13:22:37.326822   866 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:37.344875   824 raft_consensus.cc:1050] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:37.347347   824 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 13, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.460345   626 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Committing config change with OpId 20.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.465556 25621 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20240426 13:22:37.469959   866 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.476935 25030 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.477509 25104 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.490115 25105 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20240426 13:22:37.491655 25031 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20240426 13:22:37.495218 25624 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.496820   824 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.004s
I20240426 13:22:37.513152   896 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.566388 24854 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 14 to 15, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.579360   864 raft_consensus.cc:1050] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:37.584416   824 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Committing config change with OpId 27.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.586460   864 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 13, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.588958 25030 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.630116 24947 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.633826 25624 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.634337 25104 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.668815 25103 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20240426 13:22:37.650691 24847 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 27 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:37.670370   896 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.678071   866 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Committing config change with OpId 20.16: config changed from index 15 to 16, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.663822 25622 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20240426 13:22:37.657651 25030 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.681138   906 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:37.679491 25103 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Committing config change with OpId 20.16: config changed from index 15 to 16, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.686671 25622 raft_consensus.cc:2932] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Committing config change with OpId 20.16: config changed from index 15 to 16, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.702459 25621 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:37.703909   786 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.711122 25620 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:37.713131   893 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Committing config change with OpId 24.16: config changed from index 15 to 16, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.713997 25103 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:37.716987 25620 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.718647   843 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:37.734102 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:37.736071 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aca59ad77ad642c48a7e7eed1635089d with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:37.738931 25027 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:37.740111   843 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.740728   892 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.741202 25618 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:37.742820   892 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.743311 24948 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:37.750852   892 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:37.776214 25104 raft_consensus.cc:2932] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.779368 24948 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:37.810647   792 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20240426 13:22:37.832840 24846 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 15 to 16, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 20 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.840272 24956 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:37.844547 24847 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 15 to 16, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 24 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:37.861856 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59926.8us
I20240426 13:22:37.864078 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:37.867336 25624 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20240426 13:22:37.870026 24948 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.884335 25030 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20240426 13:22:37.869511   892 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.885774   896 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:37.893141 25618 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.917394 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet aca59ad77ad642c48a7e7eed1635089d with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:37.922082   896 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Committing config change with OpId 27.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.923432   906 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:37.933055 25030 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Committing config change with OpId 27.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.940280 25622 raft_consensus.cc:2932] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Committing config change with OpId 27.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:37.981681 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:37292
I20240426 13:22:38.001662 25031 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.069923   924 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:38.071153   924 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:38.063419   919 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:38.078671   924 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:38.098167 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 242110e36ca34176b9d522087d440de6 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:38.101068 24849 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.103580   924 ts_tablet_manager.cc:1875] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240426 13:22:38.130384 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60135.4us
W20240426 13:22:38.137799 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 180650us
I20240426 13:22:38.159961 25030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:38.160861 25030 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:38.168394 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95237.7us
W20240426 13:22:38.175721 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 112090us
I20240426 13:22:38.198089   924 ts_tablet_manager.cc:1888] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.15
I20240426 13:22:38.198576   924 log.cc:1196] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/5f4cb6239dd94f7693fdaacdc8ae8a8f
I20240426 13:22:38.198860 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:38.200600 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:38.201323 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:38.200603 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:38.241541   919 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:38.242458   919 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:38.250101 25103 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:38.237989 24849 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 27 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.268023   919 ts_tablet_manager.cc:1875] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:38.274801 25622 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:38.280692   928 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20240426 13:22:38.281850 25030 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:38.296272   892 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:38.301590   735 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:38.308225 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52268.8us
I20240426 13:22:38.307466 25622 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.303874 25030 raft_consensus.cc:2932] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.328156 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79bb183792fd4ae08dd4564ec82efc36 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:38.349375 24849 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 16 to 17, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:38.361676 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 242110e36ca34176b9d522087d440de6 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:58852
I20240426 13:22:38.366691 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:37292
I20240426 13:22:38.368749   924 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:38.369925   924 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:38.370316   936 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:38.370826   924 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:38.371419   936 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:38.372355   936 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:38.377048   919 ts_tablet_manager.cc:1888] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 20.15
I20240426 13:22:38.377161   924 ts_tablet_manager.cc:1875] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:38.377606   919 log.cc:1196] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/aca59ad77ad642c48a7e7eed1635089d
I20240426 13:22:38.379935 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet aca59ad77ad642c48a7e7eed1635089d (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:38.399114   843 raft_consensus.cc:979] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:38.399660   792 raft_consensus.cc:1067] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:38.410857   936 ts_tablet_manager.cc:1875] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:38.415172 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41288
I20240426 13:22:38.415828 24948 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:38.416211 24948 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:38.422740 24948 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:38.430127 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:38.430716 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:38.431294 25103 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:38.430227 24948 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:38.430747 25027 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Stepping down as leader of term 21
I20240426 13:22:38.437932 25027 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:38.438704 25027 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:38.444809   924 ts_tablet_manager.cc:1888] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.16
I20240426 13:22:38.445389   924 log.cc:1196] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/79bb183792fd4ae08dd4564ec82efc36
I20240426 13:22:38.445338 25027 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:38.447263 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 79bb183792fd4ae08dd4564ec82efc36 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:38.448519 25103 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:38.449723 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:38.450587   824 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:38.451081   824 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:38.451898   824 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:38.452479   936 ts_tablet_manager.cc:1888] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 27.16
I20240426 13:22:38.452687 25027 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 22.
I20240426 13:22:38.453228   936 log.cc:1196] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/242110e36ca34176b9d522087d440de6
I20240426 13:22:38.455127 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 242110e36ca34176b9d522087d440de6 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:38.480049 24846 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 21 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 22 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:38.995494 25027 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:39.004539   896 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:39.075866 25099 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20240426 13:22:39.077641   866 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20240426 13:22:39.314090 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:39.442494 24947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:34068
I20240426 13:22:39.443133 24947 raft_consensus.cc:595] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:39.448184 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:39.449949 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:39.458715 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:39.612565   866 raft_consensus.cc:979] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:39.613075   866 raft_consensus.cc:1067] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:22:39.614578 25030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
 from {username='slave'} at 127.0.0.1:51378
I20240426 13:22:39.615146 25030 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:39.615481 25030 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:39.626504 25030 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:39.637487 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:22:39.638108 24947 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Stepping down as leader of term 22
I20240426 13:22:39.638411 24947 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:39.638772 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:39.639437 25103 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:39.639132 24947 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:39.638551 25030 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:39.641362 24947 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:39.645907 25103 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:39.647152 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:39.647966   929 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election won for term 23
I20240426 13:22:39.648479   929 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:39.649281   929 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:39.653283 24947 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 23.
I20240426 13:22:39.676504 24948 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:39.697901 24849 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 22 to 23, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:39.713663 24920 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d8fda82f95348219809212d99d6704e. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:39.716951 25103 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:39.718582 25030 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:39.718454   896 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:39.721454   944 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:39.740284   824 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:39.748731 25099 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:39.781387 25027 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:39.784090 25099 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:39.783210 24846 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:39.834823 25027 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20240426 13:22:39.839035   735 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:39.840065 24948 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20240426 13:22:39.843654   892 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240426 13:22:39.847314 25069 consensus_peers.cc:485] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7408112bace482693c999f678aa9d73. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:39.904577 24948 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Committing config change with OpId 22.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:39.902053   892 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Committing config change with OpId 22.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:39.916198 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79172.2us
I20240426 13:22:39.976377 25031 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:39.986277 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60336.4us
I20240426 13:22:40.006903 24849 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:40.026108 25103 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:40.040797 25099 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:22:40.042655   929 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:40.055169 24948 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:22:40.069252   929 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
W20240426 13:22:40.083683 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9108945c08ba4b2287c1e238b402722f. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:40.096246 25031 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:40.099128   892 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:40.111212 24947 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20240426 13:22:40.119941   953 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20240426 13:22:40.127647   892 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:40.133049 25031 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:40.155741 24846 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:40.212074 25031 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:40.305696 24947 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.15: config changed from index -1 to 15, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:40.352890 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a69cdcf2ee4768a20888dd974c6ba7. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:40.356873   970 ts_tablet_manager.cc:902] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:40.408238 25099 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20240426 13:22:40.427098   976 ts_tablet_manager.cc:902] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:22:40.440346 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 74012.1us
I20240426 13:22:40.424294   843 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
W20240426 13:22:40.520848 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86332us
I20240426 13:22:40.598975 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:40.684219   983 ts_tablet_manager.cc:902] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:40.744824   976 tablet_copy_client.cc:308] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
W20240426 13:22:40.778929 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 263131us
W20240426 13:22:40.796221 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 371409us
I20240426 13:22:40.753583   970 tablet_copy_client.cc:308] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
W20240426 13:22:40.830504 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57373.1us
W20240426 13:22:40.874223 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 178212us
I20240426 13:22:40.962896 25027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:60586
I20240426 13:22:40.963590 25027 raft_consensus.cc:595] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:40.967016   982 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.286s	user 0.000s	sys 0.002s
I20240426 13:22:40.973944 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:40.975762 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:40.976461 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:22:40.973872 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 230112us
I20240426 13:22:40.984126 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet f7408112bace482693c999f678aa9d73 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:22:40.984747 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet f7408112bace482693c999f678aa9d73 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-f7408112bace482693c999f678aa9d73
I20240426 13:22:41.037176 25114 tablet_copy_source_session.cc:205] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
W20240426 13:22:41.093248 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 308950us
W20240426 13:22:41.165302 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 196581us
W20240426 13:22:41.199990 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 207603us
I20240426 13:22:41.241446 24947 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20240426 13:22:41.261988 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 280110us
W20240426 13:22:41.259078   983 thread.cc:636] tablet-download-pool- [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.033s	sys 0.064s
I20240426 13:22:41.299991 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 0d8fda82f95348219809212d99d6704e from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:22:41.300674 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 0d8fda82f95348219809212d99d6704e from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-0d8fda82f95348219809212d99d6704e
I20240426 13:22:41.300928   970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:22:41.296733   983 thread.cc:602] tablet-download-pool- [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.033s	sys 0.064s
W20240426 13:22:41.326447 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61072.8us
I20240426 13:22:41.385596 24965 tablet_copy_source_session.cc:205] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:41.502489   992 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:41.505052   992 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:41.543234   992 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:41.556483   980 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240426 13:22:41.573570 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 233134us
I20240426 13:22:41.580407   970 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44496 (request call id 1580) took 1253 ms. Trace:
I20240426 13:22:41.581121   970 rpcz_store.cc:276] 0426 13:22:40.326777 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:40.326899 (+   122us) service_pool.cc:225] Handling call
0426 13:22:40.357326 (+ 30427us) ts_tablet_manager.cc:903] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
0426 13:22:41.283023 (+925697us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:41.566057 (+283034us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:41.566396 (+   339us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:41.576084 (+  9688us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:41.579777 (+  3693us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:41.580238 (+   461us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:41.580315 (+    77us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55088,"tablet-copy.queue_time_us":9781,"thread_start_us":328411,"threads_started":5}
I20240426 13:22:41.583215   970 tablet_copy_client.cc:783] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:41.588266   992 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:41.589672   992 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:41.591624   992 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:41.596719  1022 ts_tablet_manager.cc:902] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:41.602304 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:41.603289 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:41.610790 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:41.613925   970 tablet_copy_client.cc:647] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
W20240426 13:22:41.627988 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46319.5us
I20240426 13:22:41.632346  1018 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Committing config change with OpId 23.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:41.639039 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54100.7us
W20240426 13:22:41.642731 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 88411.5us
I20240426 13:22:41.662734 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:41.666646   976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:41.658494 25105 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Committing config change with OpId 23.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:41.691605 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:41.713677   983 tablet_copy_client.cc:308] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:22:41.716544   980 raft_consensus.cc:979] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:41.717133   980 raft_consensus.cc:1067] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:22:41.719202 25097 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
 from {username='slave'} at 127.0.0.1:58860
I20240426 13:22:41.719980 25097 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:41.720325 25097 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:41.722676   944 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:41.732209 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:41.729825 25097 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:41.738741   977 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:41.748358 25097 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:41.751518 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:41.752113 25022 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Stepping down as leader of term 25
I20240426 13:22:41.752374 25022 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:41.753046 25022 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:41.754295 25022 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:41.772176 24948 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Committing config change with OpId 23.16: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:22:41.774544 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:41.775266 25618 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:41.779657 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 16, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:22:41.786916 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104632us
I20240426 13:22:41.791476  1018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45336 (request call id 1) took 1581 ms. Trace:
I20240426 13:22:41.791894  1018 rpcz_store.cc:276] 0426 13:22:40.209707 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:40.209882 (+   175us) service_pool.cc:225] Handling call
0426 13:22:40.224323 (+ 14441us) log.cc:842] Serialized 492 byte log entry
0426 13:22:41.791391 (+1567068us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":83104,"raft.run_cpu_time_us":1665,"raft.run_wall_time_us":47102,"thread_start_us":74426,"threads_started":2,"wal-append.queue_time_us":85095,"wal-append.run_cpu_time_us":1682,"wal-append.run_wall_time_us":1009106}
W20240426 13:22:41.798588 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117998us
I20240426 13:22:41.801420 25618 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
I20240426 13:22:41.802409 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:41.812404 25022 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
I20240426 13:22:41.823730  1020 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:41.824422  1020 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:41.824666   970 tablet_copy_client.cc:523] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:41.824748 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 9108945c08ba4b2287c1e238b402722f from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:22:41.825531 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 9108945c08ba4b2287c1e238b402722f from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-9108945c08ba4b2287c1e238b402722f
I20240426 13:22:41.825309  1020 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:41.851879   970 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:41.852826 25114 tablet_copy_source_session.cc:205] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:41.861315   976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44512 (request call id 1925) took 1501 ms. Trace:
I20240426 13:22:41.925252   983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:41.908792   976 rpcz_store.cc:276] 0426 13:22:40.360309 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:40.362414 (+  2105us) service_pool.cc:225] Handling call
0426 13:22:40.427482 (+ 65068us) ts_tablet_manager.cc:903] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
0426 13:22:41.665488 (+1238006us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:41.747913 (+ 82425us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:41.748196 (+   283us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:41.859748 (+111552us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:41.860324 (+   576us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:41.861200 (+   876us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:41.861258 (+    58us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":302,"tablet-copy.queue_time_us":64266,"thread_start_us":368454,"threads_started":5}
I20240426 13:22:41.932274   976 tablet_copy_client.cc:783] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:41.932726   976 tablet_copy_client.cc:647] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:41.967667  1022 tablet_copy_client.cc:308] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:22:41.969805 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet a6a69cdcf2ee4768a20888dd974c6ba7 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:22:41.970383 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet a6a69cdcf2ee4768a20888dd974c6ba7 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-a6a69cdcf2ee4768a20888dd974c6ba7
I20240426 13:22:41.998112 25041 tablet_copy_source_session.cc:205] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:22:42.015559  1022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:42.023630   976 tablet_copy_client.cc:523] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:42.037626   976 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:42.066655   983 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44496 (request call id 1583) took 1600 ms. Trace:
I20240426 13:22:42.073565   983 rpcz_store.cc:276] 0426 13:22:40.466696 (+     0us) service_pool.cc:168] Inserting onto call queue
0426 13:22:40.466847 (+   151us) service_pool.cc:225] Handling call
0426 13:22:40.684754 (+217907us) ts_tablet_manager.cc:903] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
0426 13:22:40.962507 (+277753us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8d3cee2e 00007fda96208789 00007fda8d3ec0eb 00007fda8d3e8e5a 00007fda8d3e8d82 00007fda952092da
0426 13:22:41.913852 (+951345us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:22:42.006435 (+ 92583us) tablet_metadata.cc:641] Metadata flushed
0426 13:22:42.006679 (+   244us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:22:42.065893 (+ 59214us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:22:42.066151 (+   258us) tablet_replica.cc:205] Creating consensus instance
0426 13:22:42.066440 (+   289us) ts_tablet_manager.cc:958] Replica registered.
0426 13:22:42.066583 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":211,"spinlock_wait_cycles":20462336,"tablet-copy.queue_time_us":190209,"thread_start_us":1049199,"threads_started":5}
I20240426 13:22:42.068835  1022 tablet_copy_client.cc:783] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:42.077623   983 tablet_copy_client.cc:783] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:22:42.107182   983 tablet_copy_client.cc:647] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:42.111212   983 tablet_copy_client.cc:523] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:42.088503  1022 tablet_copy_client.cc:647] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:22:42.135877  1022 tablet_copy_client.cc:523] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:22:42.158162  1022 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:42.158262   983 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:22:42.211973 24849 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 25 to 26, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 26 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:42.285933 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:42.387410 25621 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20240426 13:22:42.389781   735 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20240426 13:22:42.426580   970 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240426 13:22:42.439647 25026 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20240426 13:22:42.447571  1000 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20240426 13:22:42.491715   970 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:42.492950   970 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:42.567586   970 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.716s	user 0.049s	sys 0.017s
W20240426 13:22:42.630249   976 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.054s	user 0.001s	sys 0.003s
I20240426 13:22:42.666196   970 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:42.667675   970 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:42.669302   970 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:42.692008   970 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.124s	user 0.011s	sys 0.000s
I20240426 13:22:42.721940 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f7408112bace482693c999f678aa9d73 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:22:42.722532 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f7408112bace482693c999f678aa9d73 on tablet f7408112bace482693c999f678aa9d73 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:42.730046   976 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=2 applied=15 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:42.733067   976 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:42.759676 25618 raft_consensus.cc:1203] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Deduplicated request from leader. Original: 22.15->[22.16-22.16]   Dedup: 22.16->[]
I20240426 13:22:42.768335   976 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.731s	user 0.046s	sys 0.021s
I20240426 13:22:42.853683   976 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:42.859907   976 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:42.936234   976 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:42.939036   976 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.170s	user 0.007s	sys 0.001s
W20240426 13:22:42.940762 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 51772.5us
I20240426 13:22:42.962203 24965 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-0d8fda82f95348219809212d99d6704e received from {username='slave'} at 127.0.0.1:46260
I20240426 13:22:42.962702 24965 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-0d8fda82f95348219809212d99d6704e on tablet 0d8fda82f95348219809212d99d6704e with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:42.993806 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:49046
I20240426 13:22:42.994599 25098 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:42.996076 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:22:42.997823 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:22:43.010668 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:43.020606 25618 raft_consensus.cc:1203] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Deduplicated request from leader. Original: 24.14->[24.15-24.15]   Dedup: 24.15->[]
I20240426 13:22:43.064337  1022 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=1 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:43.065341  1022 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:43.067028   983 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=2 applied=15 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:22:43.072467   983 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:22:43.073189   983 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.915s	user 0.056s	sys 0.019s
I20240426 13:22:43.083873  1022 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.926s	user 0.051s	sys 0.022s
I20240426 13:22:43.090569  1022 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:43.091498  1022 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:43.093605   983 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:43.094568   983 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:22:43.095309   983 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:43.100458  1022 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:43.142598  1022 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.058s	user 0.006s	sys 0.000s
I20240426 13:22:43.200671   983 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.127s	user 0.006s	sys 0.001s
I20240426 13:22:43.206493 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-9108945c08ba4b2287c1e238b402722f received from {username='slave'} at 127.0.0.1:55910
I20240426 13:22:43.206944 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-9108945c08ba4b2287c1e238b402722f on tablet 9108945c08ba4b2287c1e238b402722f with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:43.202445 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-a6a69cdcf2ee4768a20888dd974c6ba7 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:22:43.209650 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-a6a69cdcf2ee4768a20888dd974c6ba7 on tablet a6a69cdcf2ee4768a20888dd974c6ba7 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:22:43.240753   994 raft_consensus.cc:1050] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:43.243674   994 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 14, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:43.305416 25621 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:22:43.306946   956 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:43.307559 25023 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:22:43.308818   956 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:43.319788   956 raft_consensus.cc:979] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:43.320394   956 raft_consensus.cc:1067] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:22:43.322525 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
 from {username='slave'} at 127.0.0.1:41286
I20240426 13:22:43.323112 24949 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:22:43.323434 24949 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:43.350966 24955 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20240426 13:22:43.356978 24949 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:43.357362   994 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:43.386289   956 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Committing config change with OpId 22.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.388723 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:43.389528 25023 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:43.397673 25026 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.408089 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:43.408887 25098 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Stepping down as leader of term 26
I20240426 13:22:43.409207 25098 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:43.409999 25098 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:43.411130 25098 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:43.416779 24849 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:43.427549 25621 raft_consensus.cc:1203] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEARNER]: Deduplicated request from leader. Original: 23.15->[23.16-23.16]   Dedup: 23.16->[]
I20240426 13:22:43.414146 24949 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:43.433046 25621 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Committing config change with OpId 22.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.464954 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 27.
I20240426 13:22:43.475926  1017 raft_consensus.cc:1050] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:43.504917 25023 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 27.
I20240426 13:22:43.506181 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:43.507519  1021 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:43.508250  1021 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:22:43.509143  1021 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:43.510923 24950 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Committing config change with OpId 22.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.520309  1017 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 13, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:43.554627 24854 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 26 to 27, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 27 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:43.558069 25618 raft_consensus.cc:1203] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Deduplicated request from leader. Original: 24.14->[24.15-24.15]   Dedup: 24.15->[]
I20240426 13:22:43.621812 25624 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:43.624786  1006 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:43.656014 25023 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:43.662824  1017 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:43.688191 25097 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:43.697595  1017 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:43.718443  1017 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.732882 25023 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.736392 25104 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.744050 25621 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.801661 25097 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:43.817333 25621 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20240426 13:22:43.851437 24854 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.860170  1026 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20240426 13:22:43.863629 25023 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20240426 13:22:43.877885   956 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Committing config change with OpId 22.18: config changed from index 17 to 18, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.882239   994 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240426 13:22:43.882941 25621 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Committing config change with OpId 22.18: config changed from index 17 to 18, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.896212 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7408112bace482693c999f678aa9d73 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:43.898634 25026 raft_consensus.cc:2932] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Committing config change with OpId 22.18: config changed from index 17 to 18, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.909271 24950 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:43.918640 25618 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:43.925763  1054 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240426 13:22:43.942863 24840 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet f7408112bace482693c999f678aa9d73 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER failed with leader 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 17 but the committed config has opid_index of 18
I20240426 13:22:43.946673   994 raft_consensus.cc:1050] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:43.949532   994 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 13, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:43.957549 24854 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 17 to 18, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 22 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.962740 25023 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20240426 13:22:43.965534  1017 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.981997 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d8fda82f95348219809212d99d6704e with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:43.985338  1021 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:43.993781 25618 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:43.994308 24854 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 24 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:43.998575 25023 raft_consensus.cc:2932] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.001778 25026 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20240426 13:22:44.030189  1017 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:44.036995 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8fda82f95348219809212d99d6704e with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:58852
I20240426 13:22:44.047477 25098 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20240426 13:22:44.049108  1017 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240426 13:22:44.061425 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet f7408112bace482693c999f678aa9d73 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:37292
I20240426 13:22:44.063514 24950 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:44.063813 25621 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20240426 13:22:44.074584   735 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:44.097110   994 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:44.127367  1072 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:44.128646  1072 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:44.192443  1073 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:44.222190 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:22:44.201682  1072 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:44.308667  1076 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20240426 13:22:44.308732 25026 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20240426 13:22:44.315294 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 85930.6us
W20240426 13:22:44.342363 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 82374.2us
W20240426 13:22:44.435484 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 93372.1us
I20240426 13:22:44.419389  1073 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:44.483479 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49427.2us
W20240426 13:22:44.505118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 159603us
I20240426 13:22:44.509929  1072 ts_tablet_manager.cc:1875] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240426 13:22:44.512091 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79423.3us
I20240426 13:22:44.505802  1073 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:44.556295  1026 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240426 13:22:44.554469 24949 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.564314  1073 ts_tablet_manager.cc:1875] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
/home/jenkins-slave/workspace/build_and_test@3/src/kudu/master/auto_leader_rebalancer-test.cc:427: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 1
I20240426 13:22:44.572235 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
W20240426 13:22:44.591799 25069 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:44.550753   956 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.633641  1010 raft_consensus.cc:1050] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:22:44.635990 25618 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:44.643568 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 171976us
W20240426 13:22:44.653997 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.725553  1072 ts_tablet_manager.cc:1888] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.16
I20240426 13:22:44.726132  1072 log.cc:1196] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/0d8fda82f95348219809212d99d6704e
I20240426 13:22:44.754010 24847 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:44.764339 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.770047 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44116.5us
I20240426 13:22:44.774834 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 0d8fda82f95348219809212d99d6704e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:44.783321 25023 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Committing config change with OpId 24.16: config changed from index 15 to 16, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.818358  1010 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:44.828127  1073 ts_tablet_manager.cc:1888] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 22.17
I20240426 13:22:44.828678  1073 log.cc:1196] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/f7408112bace482693c999f678aa9d73
I20240426 13:22:44.830314 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet f7408112bace482693c999f678aa9d73 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
W20240426 13:22:44.830626 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.833690 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.843075 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.848208 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.848862 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.849603 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.850190 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.850806 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.851487 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.852224 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.852880 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.853556 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.859182 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60218.4us
W20240426 13:22:44.880656 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.888876 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.892140 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.894915 25105 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20240426 13:22:44.895104 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.899225 25618 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20240426 13:22:44.898768 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.901017  1010 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
W20240426 13:22:44.905922 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.909533 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.910607  1104 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
W20240426 13:22:44.919026 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.924908 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.927137 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.940552 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.950642 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:44.959137 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:44.991379  1104 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:44.993305 25622 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.033598 25101 raft_consensus.cc:2932] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Committing config change with OpId 23.17: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.068442 25105 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.088595 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 16 to 17, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 23 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:45.092676 25622 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20240426 13:22:45.099251 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.100855  1097 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
W20240426 13:22:45.111652 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.164762  1094 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.165541 25622 raft_consensus.cc:2932] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:22:45.181263 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:45.188351  1080 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.188946  1080 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.190910  1080 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.193002 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.194702 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.196800 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.198643  1080 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.211783 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9108945c08ba4b2287c1e238b402722f with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:22:45.212304  1083 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.212860  1083 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.215037  1083 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.217613 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.221398 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.223385 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.224578  1085 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.239740  1085 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.241562  1085 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.243409  1085 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.249439 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.251071 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.265107 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.266422  1083 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.287696 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:45.284451 24846 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 16 to 17, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:45.294436 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.296803 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.298807 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.304309 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.305140 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.305745 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.313620 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.315790  1100 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.318962 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.320134 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.316445  1100 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.322733 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.323273  1100 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.323606 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.324975 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.325974 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.329408 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9108945c08ba4b2287c1e238b402722f with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:34690
I20240426 13:22:45.327564 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.338201  1118 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:45.378266 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.379112  1085 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.379773  1085 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.380990 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.381893  1085 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.382764 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.383980  1119 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:45.381745 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.385051  1119 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.385144 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.386487 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.388260 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.389183 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.389864 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.391989  1083 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:45.389977 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.399096  1119 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.403944  1119 ts_tablet_manager.cc:1875] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:22:45.405019 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.407445 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.408450  1083 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.409404 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.409019  1083 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.414006  1085 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.415117  1085 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.419076 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.420521 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.422165 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.422422  1085 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.422581 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.423080  1106 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:45.423939 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.424340  1083 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.424939 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.425757 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.427026 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.428485 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.429797  1106 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:45.430470  1106 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.430990  1106 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:22:45.432276 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.433069  1106 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.434535 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.437714  1085 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.440475  1085 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.434535 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.443850 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.444875  1085 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.445744 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.445755 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.448004 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.449033  1106 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20240426 13:22:45.445452  1085 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:45.450843 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.451736 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.467657  1106 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.468294  1106 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.470352  1106 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.476539 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.479945 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.482333  1119 ts_tablet_manager.cc:1888] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 24.16
I20240426 13:22:45.482841  1119 log.cc:1196] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/9108945c08ba4b2287c1e238b402722f
I20240426 13:22:45.483029  1083 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.483556  1083 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.484277 24920 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:45.485433  1083 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.485944  1083 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20240426 13:22:45.487643  1118 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.488149  1118 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.490024  1118 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.490128 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.491389 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.492630 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.487098 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 9108945c08ba4b2287c1e238b402722f (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:22:45.495515 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.495527 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.496901  1085 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20240426 13:22:45.497200 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.498289  1085 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:45.459620 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.502056 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.514619 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.515748  1118 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:22:45.516309  1118 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:45.517678 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.518210  1118 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:45.518698 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.519407 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.523546 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.531911  1118 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.532496  1118 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.539445 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:45.536440 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.547163  1118 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.546465 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.582475 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.583776  1118 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:45.586220 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.587163 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.587814 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.650810 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.599182 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:45.652664 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 53558.3us
I20240426 13:22:45.651813 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.669633 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.672371 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.675094  1118 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240426 13:22:45.678503 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.679111 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.686054 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.686882 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.687768 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.691609 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.692402 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.692926 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.695467 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.719192 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.722203 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.729033 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.729899 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.735805 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.742772 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.743548 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.744129 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.752629 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.756510 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.757210 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.761301 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.762115 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.763177 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.799274 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.800017 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.801122 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.803732 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.806157 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.825973 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.834414 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.835345 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.836014 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.847889 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.848788 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.849542 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.852772 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.853610 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.854187 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.855971  1085 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:45.856544  1085 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.857007 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.857877 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.858422 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.858503  1085 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.860991 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.862632 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.863318  1085 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:45.863451 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.864266 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.863986  1085 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:45.867025 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.867461 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:45.867847 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.868592 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.866278  1085 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:45.871915 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.874307 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.877468 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.881847 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.889531 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.890764 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.894011 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.894701 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.900086 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.909188 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.906903 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.910369  1085 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:45.912587 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.913401 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.914062 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 17 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.895584 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:45.915953 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:45.916950  1118 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:45.925469 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.926429 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.926999 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.930102 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.937103 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.937781 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.942082 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.946494 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.947643 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.950217 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.950990 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.951675 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.966591 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.967413 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.968502 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.977528 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.978313 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.978906 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.985679 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.988626 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 LEADER]: Raft consensus shutting down.
I20240426 13:22:45.989727 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 18 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.992524 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.993221 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:45.993755 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:45.996222 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:45.996845 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.007515 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.013038 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.015576 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.017992 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.021683 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.026161 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:46.031155 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.034119 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.034842 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:46.044942 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.048658 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:22:46.049250 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:46.049822 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:46.390157  1125 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.390718  1125 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.392827  1125 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:46.396026 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:46.396272 25069 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:46.397589 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.400100  1122 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:46.651064  1124 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.651690  1124 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.653949  1124 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.658227 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.658947 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 20.
I20240426 13:22:46.660270 25589 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:46.663486  1121 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:46.663893  1121 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:46.664503 25588 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.664250  1121 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:46.681870  1010 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.682533  1010 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.688669  1121 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.692646  1010 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:46.703469 25588 leader_election.cc:334] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.704640 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:46.705597 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:46.706224 25105 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:46.712138  1121 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.714277 25105 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:46.723121 25589 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.724983  1122 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.725534  1122 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.724529  1121 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:46.728843  1122 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.729410  1121 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:46.729887 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.731367  1121 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.760318  1130 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.760903  1130 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.763398  1130 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.770236  1124 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.770845  1124 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.773355  1124 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.775213 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.777459 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.781038 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
W20240426 13:22:46.779714 25588 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.786375  1084 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.786896  1084 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.790020 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.791589  1084 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.795672 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.797032  1084 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:46.797439  1084 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.797849  1084 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:46.798106 25588 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.798878  1130 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.799490  1130 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.801393  1130 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.802259  1122 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.802853  1122 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.805846 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.807442  1122 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.807894 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:22:46.810415 25069 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.810879 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:46.811908  1026 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:46.809312  1124 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20240426 13:22:46.813400 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:46.813513 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:46.814033 25098 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:46.834709  1084 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.839669  1010 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.840287  1010 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.842340  1010 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.851601 24849 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 21 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:46.854681  1084 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.859118 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.859252  1010 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.859853  1010 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.861845  1010 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:46.866957 24995 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.867458 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.868721  1130 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:46.869158  1130 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.869525  1130 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:46.872233 25588 leader_election.cc:334] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.877481  1010 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:46.877142  1130 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.878631  1122 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.879388  1122 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.884229 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:46.884867 25023 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
I20240426 13:22:46.887671 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.888589  1026 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:46.888978  1026 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.889303  1026 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:46.890239 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.896531 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:46.897258 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
I20240426 13:22:46.880208  1130 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.900482 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:46.901088 25105 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:46.904275 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:46.905965 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:46.907619 25022 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:46.911576 25069 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.915279 25022 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:46.916435  1122 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:46.923249  1026 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:46.926561 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:46.927284 25022 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:46.935078 25098 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 24: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:46.949766 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:46.950626  1010 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:46.951110  1010 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.951550  1010 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:46.912091 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:46.954414  1122 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:46.955446 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43894.9us
I20240426 13:22:46.959931  1010 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:46.964053 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.984791 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:46.985639  1104 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20240426 13:22:46.987452 24995 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:46.981695 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:46.988685  1121 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:46.989190  1121 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:46.990093  1121 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.981096  1084 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:46.991667  1084 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:46.994865 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:47.001405 25588 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.001953 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.002784  1124 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20240426 13:22:47.003731  1084 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:46.981928 25105 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
I20240426 13:22:46.965620  1026 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.006170 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.006951  1130 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:47.007675  1130 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.008745  1130 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.014861 25069 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.016923  1010 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.018649 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.037293 25105 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.025817  1122 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.038620  1122 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.039548   994 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.040210   994 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.042063   994 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.042383  1122 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.043244 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.043989 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 25.
I20240426 13:22:47.046207 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.047019   994 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20240426 13:22:47.047461   994 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.047041 24846 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 25 to 26, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 26 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.047870   994 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Advancing to term 26
W20240426 13:22:47.052207 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.063496   994 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.069772 25069 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.072741 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.075223 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.075946 25619 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 18.
W20240426 13:22:47.077858 24995 leader_election.cc:334] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.093431 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.081908 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.108371 25028 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Advancing to term 26
I20240426 13:22:47.032207 25022 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 24.
I20240426 13:22:47.112074 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.112828  1130 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:47.089638   994 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.094635  1125 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.114738  1026 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:47.114830  1125 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.115576  1026 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.116618  1026 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.117149  1125 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.079041 25069 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.104187  1010 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.123250  1010 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.105408 24849 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 24 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.133239  1010 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.104749 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.148249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43542.8us
W20240426 13:22:47.135784 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.138065 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.106793 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
I20240426 13:22:47.154770  1122 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:47.155236  1122 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.155629  1122 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.159262 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:22:47.161624 25069 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.162616 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.164268  1122 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.164460 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.167006 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.155488 24995 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.167382  1130 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:47.168841  1130 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.169724  1130 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.177000  1122 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.188897 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.192643 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 26.
W20240426 13:22:47.197515 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.200984 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.201920  1122 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader election won for term 26
I20240426 13:22:47.202646  1122 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.203440  1122 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.204584 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.205246 25619 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.215480  1084 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.216109  1084 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.218113  1084 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.224606  1121 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.223766 24849 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 24 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 25 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.225153  1121 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.225786  1084 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.226850  1084 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.229255  1084 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.231071  1121 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.189028 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.234270 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45268.3us
W20240426 13:22:47.242152 25588 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.249349  1121 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.249640 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.249930  1121 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.250597 25105 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 26.
I20240426 13:22:47.253131 25589 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.254474  1084 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:22:47.254956  1084 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.255367  1084 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:47.256071   994 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.257012  1125 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.257485  1125 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.259409  1125 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.256599   994 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.262022   994 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.264837  1121 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.270851 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.284587  1124 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.304980  1124 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.304572 25619 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
W20240426 13:22:47.287608 25069 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.291636  1130 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.309900  1130 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.311709 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.311977  1130 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.312800 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.314692 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.320334 25588 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.321197 25105 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20240426 13:22:47.313226 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.323204  1122 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.323871  1122 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:47.323277 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.325457 24995 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.325824  1122 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.325945 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:47.299993 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101265us
I20240426 13:22:47.306975  1124 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.286113 24846 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 23 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 24 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:22:47.326566 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9960 after lost signal to thread 23799
W20240426 13:22:47.330992 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d99e0 after lost signal to thread 24888
I20240426 13:22:47.331496  1130 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20240426 13:22:47.332588 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9f60 after lost signal to thread 25055
W20240426 13:22:47.330916 25069 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.333889 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9540 after lost signal to thread 25129
I20240426 13:22:47.333976  1104 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.336570  1104 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.338907  1104 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.340371  1124 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:47.341040 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08001a7900 after lost signal to thread 25650
I20240426 13:22:47.340911  1124 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.344962  1124 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.348228 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.349086  1084 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.353034  1124 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240426 13:22:47.361093 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.380769 25617 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 19.
I20240426 13:22:47.361234 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.363015 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.365065  1084 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.366183 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.385628 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.383100  1130 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.386265 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:47.386590  1130 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.387310 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.388535  1130 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.390239 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.390463  1010 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.381803 25615 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:47.390899  1010 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.371541  1122 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:47.372709 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.392406 25030 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:47.373724 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.375360 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.394001 25022 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 18.
I20240426 13:22:47.384491 24846 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 25 to 26, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 26 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.382603 25028 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 27.
I20240426 13:22:47.394443  1010 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.394966  1010 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20240426 13:22:47.395243  1010 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.395527  1010 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Advancing to term 28
W20240426 13:22:47.396078 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.401098 25588 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.401479 25105 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:47.415341 25105 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 27.
I20240426 13:22:47.416085 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.417044 25026 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:47.421221 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.421964 25617 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 22.
I20240426 13:22:47.422932 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:47.435133 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.421914 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.436313 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.439299  1125 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20240426 13:22:47.441046 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64000.1us
W20240426 13:22:47.441659 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:47.442333 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.442613 25589 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
W20240426 13:22:47.442947 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.440447  1010 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.444272  1125 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.444701  1125 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:47.445171  1124 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:47.445708  1124 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:47.446606  1124 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.439658  1026 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:47.448455  1026 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.448721  1142 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20240426 13:22:47.449146  1142 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.449235  1026 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.450400 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.450851 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.451586 24995 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.449826  1142 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:47.453011 25028 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 28: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.453871 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.450899 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.456599   990 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20240426 13:22:47.457363 25588 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.458099 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.458788 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 64253.2us
I20240426 13:22:47.459978  1010 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:47.461052  1010 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:47.462237 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.458127  1104 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.464170 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:47.464969 25589 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.461607  1010 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.467103   990 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:47.463862  1121 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20240426 13:22:47.469702  1121 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.470060  1121 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:47.472903 25589 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.467990  1010 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Advancing to term 23
I20240426 13:22:47.473928  1125 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.479863 25098 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 21: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:22:47.472368  1104 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.482353  1125 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:47.485216 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.489640  1104 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.493551  1142 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.495422 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57991.5us
I20240426 13:22:47.483448  1121 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.506052 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.509219 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:47.510109 25098 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Advancing to term 28
W20240426 13:22:47.514837 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51506.2us
W20240426 13:22:47.516213 24995 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.517132 24995 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.517714 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.520272 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.520889 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:22:47.523105 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:47.520916 25617 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 24.
W20240426 13:22:47.523869 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.524452 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.524966 24995 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.526791 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.508599  1142 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.541373 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:47.538674  1010 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.547391  1145 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20240426 13:22:47.535923 23796 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:47.561497 25069 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.509094   990 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20240426 13:22:47.562420  1010 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:47.563159 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 98275.9us
I20240426 13:22:47.531582  1084 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20240426 13:22:47.568228 25069 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.570785 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.571259 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.571728 25069 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.572862 25069 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.564179   990 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.576798   990 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.580987 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 28.
I20240426 13:22:47.574860 24846 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 18 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 19 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:47.584247  1151 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:47.584697  1151 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.585036  1151 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.547943  1145 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.586361  1145 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:47.583657   990 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.594108 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:47.597352 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 69021.9us
I20240426 13:22:47.601694 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:47.603654  1130 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:47.640970  1156 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:47.641561 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.644776  1145 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.648581  1145 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.572635  1142 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.594146  1151 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.572654  1084 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:22:47.660303 25588 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.666203 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 139495us
W20240426 13:22:47.666900 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.667449 24995 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.674597 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.654664   990 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.682228 25617 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 23: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:47.650583  1157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election won for term 28
I20240426 13:22:47.651322  1130 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:47.684877 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.685647 25619 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:47.666777  1084 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Advancing to term 28
I20240426 13:22:47.657824  1121 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.659621  1151 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.703684 25619 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 25.
W20240426 13:22:47.720978 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 126945us
I20240426 13:22:47.652372  1142 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:47.725584 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 213269us
I20240426 13:22:47.720172 24846 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 26 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 27 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:47.734434 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.760866 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.765120 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.766008 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 23: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.765864  1084 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.768916 25027 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20240426 13:22:47.771344  1142 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:47.773447 25615 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 28: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:22:47.773466 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.774400 25030 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 18 FOLLOWER]: Advancing to term 19
I20240426 13:22:47.778461 25023 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20240426 13:22:47.778992 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.779518 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:47.781131  1084 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:47.791013 25098 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20240426 13:22:47.800457 25023 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20240426 13:22:47.798053 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52148.2us
I20240426 13:22:47.807170 25022 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
I20240426 13:22:47.809711 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 19.
I20240426 13:22:47.810777 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:47.811390 25023 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 28: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:22:47.816637 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.830961 25589 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.831838 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.832026 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 235182us
W20240426 13:22:47.833490 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.835281  1125 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:47.835901  1125 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:47.836863  1125 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.824980  1145 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20240426 13:22:47.815339  1147 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20240426 13:22:47.816893  1157 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.842193  1122 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:47.847139 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:47.847707 25615 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Advancing to term 20
I20240426 13:22:47.823702  1084 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:47.851578  1143 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader election won for term 19
I20240426 13:22:47.852123  1143 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:47.836184  1124 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20240426 13:22:47.853183  1143 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.859438 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:47.859728 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:47.860201 25619 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 26.
I20240426 13:22:47.870064 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
W20240426 13:22:47.870818 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.871417 25069 leader_election.cc:334] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.872481 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.872825  1010 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Leader election won for term 25
I20240426 13:22:47.881889 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.882723  1010 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:47.885298 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:47.886154  1010 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:47.879240  1121 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20240426 13:22:47.843973  1157 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:47.892760  1166 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20240426 13:22:47.886824  1161 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20240426 13:22:47.909265 24995 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:47.909165  1166 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:22:47.911343 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:47.922973 25615 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
I20240426 13:22:47.924584  1161 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240426 13:22:47.920943  1166 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Advancing to term 27
W20240426 13:22:47.935683 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.947427 25069 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.948298 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.951310 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.955298 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:47.958271 24995 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:47.963181 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:47.963464 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:47.952844 24846 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:22:47.964310 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:22:47.977113 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92826.4us
I20240426 13:22:48.010300  1146 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader election won for term 20
W20240426 13:22:48.050143 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.054149  1146 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:48.055325  1146 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:48.072680  1171 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20240426 13:22:48.077335  1170 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20240426 13:22:48.086481 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.111093 24849 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 18 to 19, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 19 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:22:48.129063 25098 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20240426 13:22:48.147572 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104881us
I20240426 13:22:48.152639  1166 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:48.162236  1166 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:48.148905  1154 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20240426 13:22:48.170400 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:48.171039 25617 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Advancing to term 27
I20240426 13:22:48.176697 25619 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20240426 13:22:48.176873 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.182552  1165 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20240426 13:22:48.183697  1173 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20240426 13:22:48.194216 25617 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 27.
I20240426 13:22:48.197628 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 
I20240426 13:22:48.201418 25101 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20240426 13:22:48.220739  1158 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20240426 13:22:48.204357  1104 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Leader election won for term 27
I20240426 13:22:48.225822  1104 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:22:48.226758  1104 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:48.283342 24854 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 27 to 28, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 28 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.318696 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:48.319275 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:48.422670 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80343.9us
W20240426 13:22:48.426178 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66793.8us
I20240426 13:22:48.426463 25022 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20240426 13:22:48.432453 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:48.440608 25022 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20240426 13:22:48.442229  1026 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240426 13:22:48.449182 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.463960  1176 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20240426 13:22:48.480574 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.483254 25104 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20240426 13:22:48.485672 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.489725  1158 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240426 13:22:48.494452 25619 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20240426 13:22:48.505865 25619 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20240426 13:22:48.507797  1163 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20240426 13:22:48.511231  1150 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20240426 13:22:48.526867 24846 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 19 to 20, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 20 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.559273 23796 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:48.563249 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:48.615526 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.640055 24846 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 24 to 25, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 25 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:22:48.689522 24849 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 26 to 27, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 27 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:22:48.771582 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:48.841828 25617 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20240426 13:22:48.843741  1158 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240426 13:22:49.422816 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.428328 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.513195 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.554678 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.559778 23796 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:49.568109 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.575170 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.605594  1176 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:49.606118  1176 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.607985  1176 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:49.610713 25588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111) [suppressed 397 similar messages]
W20240426 13:22:49.629650 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:49.645274 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.648419 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:49.649111 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 23.
I20240426 13:22:49.652993 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.655066  1176 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20240426 13:22:49.655472  1176 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:49.655779  1176 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Advancing to term 24
W20240426 13:22:49.662966 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.663619  1176 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.679800  1176 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:49.683408 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:49.684134 25022 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Advancing to term 24
I20240426 13:22:49.690914 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
W20240426 13:22:49.692325 25588 leader_election.cc:334] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.696507 25589 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.697721  1176 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader election won for term 24
I20240426 13:22:49.698493  1176 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:49.699388  1176 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.731081 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.741080 24846 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 22 to 24, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 24 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:49.825793 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.839496 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.847167 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.863714  1176 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:22:49.864326  1176 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:49.866289  1176 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:49.869601 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:22:49.870249 25028 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 28.
W20240426 13:22:49.870450 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.873701 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.879549  1176 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20240426 13:22:49.880000  1176 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:22:49.880378  1176 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 28 FOLLOWER]: Advancing to term 29
W20240426 13:22:49.879555 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.887359 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.904927 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.907675 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.920349 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.910969  1176 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.911649 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.929868 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:49.925323  1176 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:49.933714 25588 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:49.935551 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:22:49.936174 25022 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Advancing to term 29
W20240426 13:22:49.937963 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:49.942744 25022 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 29.
I20240426 13:22:49.944955 25589 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:22:49.946118  1176 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader election won for term 29
I20240426 13:22:49.947417  1176 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:49.948817  1176 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:49.970552 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:49.985607 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.025058 24846 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 27 to 29, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 29 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:50.086858 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.095738 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.125599 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.136864 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.146849 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.170759 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.229094 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:50.272634 25022 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20240426 13:22:50.274361 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.275416  1208 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240426 13:22:50.305732 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:50.331508 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.337718 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:50.404390 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240426 13:22:50.441936 25027 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20240426 13:22:50.443778  1208 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20240426 13:22:50.468705 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:50.503435 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.560295 23796 thread.cc:536] Waited for 4000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:50.575412 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.616986 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.684512 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.828544 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:50.894778 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.006484 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66299.8us
W20240426 13:22:51.048838 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 116625us
W20240426 13:22:51.054015 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.288707 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 194448us
W20240426 13:22:51.318878 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.331403 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.333138 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.504374 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 165815us
W20240426 13:22:51.525668 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:51.555200 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 190156us
W20240426 13:22:51.560860 23796 thread.cc:536] Waited for 5000ms trying to join with diag-logger (tid 24977)
I20240426 13:22:51.763971 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033334
I20240426 13:22:51.929883 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.165s	user 0.040s	sys 0.011s Metrics: {"bytes_written":8730,"delete_count":0,"lbm_write_time_us":13340,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":42763,"threads_started":1,"update_count":341,"wal-append.queue_time_us":42398}
I20240426 13:22:51.945315 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.440299
W20240426 13:22:51.939864 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.112576 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.125286 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.130538 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d5a0 after lost signal to thread 23799
W20240426 13:22:52.131749 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d4c0 after lost signal to thread 24888
W20240426 13:22:52.133199 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080035ac00 after lost signal to thread 25055
W20240426 13:22:52.134429 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d100 after lost signal to thread 25129
W20240426 13:22:52.135696 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08003dbd80 after lost signal to thread 25650
W20240426 13:22:52.154394 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.171723 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.267086 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.341208 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:52.340409 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.394s	user 0.126s	sys 0.037s Metrics: {"cfile_cache_miss":10,"cfile_cache_miss_bytes":29850,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":32379,"dirs.run_cpu_time_us":617,"dirs.run_wall_time_us":4899,"lbm_read_time_us":3574,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":25,"lbm_write_time_us":2952,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":11,"reinsert_count":0,"thread_start_us":47390,"threads_started":2,"update_count":369}
I20240426 13:22:52.343142 24843 maintenance_manager.cc:419] P e2ab89ff7e18462eb10e19320ff8beaa: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20240426 13:22:52.344944 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.355780 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:52.391683 24842 maintenance_manager.cc:643] P e2ab89ff7e18462eb10e19320ff8beaa: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
W20240426 13:22:52.454952 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.513686 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.534705 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.555557 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.561619 23796 thread.cc:536] Waited for 6000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:52.590241 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.593245 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.624650 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.610005 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.647534 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.648182 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.656507 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.667943 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.705339 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.746249 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.877545 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.913552 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:52.915097 24995 consensus_peers.cc:485] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.926405 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.927022 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:52.950424 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:52.994331 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:53.014695 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.016145 24995 consensus_peers.cc:485] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.060734 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.126049 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.128130 24995 consensus_peers.cc:485] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.179775 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:53.215186 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.277762 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.354686 24995 consensus_peers.cc:485] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.554603 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.563705 23796 thread.cc:536] Waited for 7000ms trying to join with diag-logger (tid 24977)
W20240426 13:22:53.590504 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.605877 24995 consensus_peers.cc:485] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.677711 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.701898 24995 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:53.798374 24995 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:54.124171 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:54.128654 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47751.4us
W20240426 13:22:54.381165 24995 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.417551 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.507091 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.551371 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.566098 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.612005 25069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111) [suppressed 443 similar messages]
W20240426 13:22:54.777138 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.787890 25069 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:54.816329 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
W20240426 13:22:54.824988 24995 consensus_peers.cc:485] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.846951 25069 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.862830 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:54.958529 24995 consensus_peers.cc:485] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.003166 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.042971 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.049167 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.058523 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
W20240426 13:22:55.061870 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.082975 25589 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.085101 25070 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.087411 24901 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240426 13:22:55.087817 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240426 13:22:55.095746 25589 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.097926 25589 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.103075 25070 consensus_peers.cc:485] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.121214 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.124385 25589 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.135139 25070 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.140012 25070 consensus_peers.cc:485] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.152022 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.162460 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240426 13:22:55.168269 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.168433 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.169554 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:55.170177 24995 consensus_peers.cc:485] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.171006 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.176803 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.178002 25589 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.177758 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.179658 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.181607 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.182196 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.182682 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.184803 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.185369 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.185914 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.189677 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.190430 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.193840 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.193862 25589 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.195045 24995 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.196784 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.197578 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.198145 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.200352 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.200959 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.201894 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.204681 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.205557 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Raft consensus shutting down.
W20240426 13:22:55.206100 25589 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.206629 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.208681 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.209336 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.210050 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.212396 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.213172 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.214251 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.216740 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.217547 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.218178 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.239518 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.241392 25070 consensus_peers.cc:485] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.241392 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:55.253221 25070 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.253367 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.253912 25069 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.255276 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.257723 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.258371 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.259305 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.267235 24995 consensus_peers.cc:485] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.267294 25070 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.270280 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.271188 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.271764 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.273480 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.274966 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.275804 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.277973 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 29 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.279906 25070 consensus_peers.cc:485] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.281563 25070 consensus_peers.cc:485] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.282852 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
W20240426 13:22:55.283769 25070 consensus_peers.cc:485] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.283735 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.284788 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.287743 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.289109 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.289727 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.290436 25069 consensus_peers.cc:485] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.292125 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.292728 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.293507 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.295959 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.296618 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.297220 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.299552 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.300261 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.300900 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.303443 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.304195 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.305366 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.307932 25589 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.308105 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.308912 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.309430 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.312089 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.312824 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.313452 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.316272 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.317031 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.317991 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.320739 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.321452 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.322613 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.325145 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.325876 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.326611 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.329183 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.329892 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.331071 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.333529 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.334218 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.335268 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.337698 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.338387 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.339850 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.342514 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.343215 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.343860 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.345399 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:55.346393 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.347661 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.348749 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.351295 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.352000 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.352612 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.355440 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.356174 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.357280 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.359787 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.360587 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.361400 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.364111 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.364898 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.365617 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 17 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.368221 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.368897 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.369480 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.372561 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.373253 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.374493 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 28 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.377146 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.377792 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.378710 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.380234 25589 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:55.381242 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.382053 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.383265 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.385815 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.386488 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.387130 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.389748 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.390415 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.391458 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.394034 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.394690 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.395718 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.397523 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.398419 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.399207 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:55.399749 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.402374 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.403018 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.405241 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.407961 25069 consensus_peers.cc:485] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:55.409482 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.412611 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.413786 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:55.416317 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:22:55.416954 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:55.418037 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:55.418037 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:55.419570 25070 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.420727 25070 consensus_peers.cc:485] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.432413 25069 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.445542 25070 consensus_peers.cc:485] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.458009 25069 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:55.474308 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.490551 25070 consensus_peers.cc:485] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.543438 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.552938 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240426 13:22:55.568991 25070 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:55.606221 25069 consensus_peers.cc:485] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:55.664830 25069 consensus_peers.cc:485] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:55.750147 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:56.208699 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.221256 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.221609 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.228353 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.228685 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.237027 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5b1db66b7c1043e1bd43d7254a4dec62, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.237392 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5b1db66b7c1043e1bd43d7254a4dec62, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.247519 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.247857 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.254961  1221 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.255630  1221 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.257695  1221 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.260210 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:56.264151 25589 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.264626 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.265616  1221 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20240426 13:22:56.274752  1221 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.275303  1221 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.277487  1221 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.281030 25588 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.282740 25589 leader_election.cc:334] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.283138 25589 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.283800  1221 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20240426 13:22:56.315795 25069 consensus_peers.cc:485] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.340577  1221 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.341293  1221 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.343662  1221 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.348667 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:22:56.349577 25589 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.351888 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.353331  1221 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:56.367038  1247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.367566  1247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.369514  1247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.372997 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.373665 25622 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 20.
W20240426 13:22:56.374845 25070 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.376094 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.377229  1247 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20240426 13:22:56.377631  1247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.370874  1221 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.378037  1247 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Advancing to term 21
I20240426 13:22:56.380599  1221 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.382854  1221 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.389921 25588 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.393209 25589 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.393682 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.395493  1221 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:56.400321  1216 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.400835  1216 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.402886  1216 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.403085  1247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.405189  1247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.412384 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:56.413059 25621 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Advancing to term 21
W20240426 13:22:56.413244 25070 leader_election.cc:334] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.413758 25070 leader_election.cc:334] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.417088 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:22:56.419667 25621 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 21.
W20240426 13:22:56.420517 25069 leader_election.cc:334] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.420893 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.421460  1247 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20240426 13:22:56.421993 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.422746  1247 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader election won for term 21
I20240426 13:22:56.423215  1247 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:56.424203  1247 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.432740 25069 consensus_peers.cc:485] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240426 13:22:56.438757 25069 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.445848  1247 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.446558  1247 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.450695 24846 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 20 to 21, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 21 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:56.451642  1216 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.452257  1216 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.454207  1216 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.454638  1247 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.456362  1247 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.456852  1221 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.456892  1247 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.457464  1221 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.459818  1247 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.462865 25070 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.466444 25070 leader_election.cc:334] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.477201  1221 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.477861 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.480079 25589 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.481488 25070 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.483209 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.483706 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.480569 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.488076  1221 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20240426 13:22:56.491050 25069 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.493240 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.493969  1216 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:56.494511  1216 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.495568 25069 leader_election.cc:334] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.495807  1221 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.496016 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.497730  1247 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.498682  1221 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.500157  1243 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.501029  1221 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.500845  1243 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.503418  1243 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:56.505976 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:56.506718 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 24.
I20240426 13:22:56.508999 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:22:56.509665 25098 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 21.
I20240426 13:22:56.513554 25589 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:56.514458  1243 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20240426 13:22:56.514935  1243 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.515348  1243 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Advancing to term 25
I20240426 13:22:56.524107 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 
I20240426 13:22:56.525149  1221 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20240426 13:22:56.525574  1221 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.525974  1221 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:56.538985  1247 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.534440  1221 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.539541  1247 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.541368  1221 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.535733 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.541788  1247 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.542502 25589 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.523211  1243 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.547808  1221 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.548431  1221 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.551504 25070 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.552326 25069 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.552170 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:22:56.552778 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.552947 25098 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Advancing to term 22
I20240426 13:22:56.554411  1216 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20240426 13:22:56.556253 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
I20240426 13:22:56.561044 25098 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 22.
I20240426 13:22:56.567248  1254 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.568215  1254 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.570370  1254 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.575685 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.578138 25588 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:56.579733  1243 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.583331 25589 leader_election.cc:334] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.584589 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.587904  1216 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.588398  1216 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.591380  1254 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader election won for term 22
I20240426 13:22:56.594082  1221 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:56.594187 25069 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.591441  1216 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:56.601097 25589 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.602293 25070 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.602346 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.604854 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.605665  1243 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.605266 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.607007  1216 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.607944  1247 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.608522  1247 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.614324  1254 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:22:56.615317  1254 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.616290  1216 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.616832  1216 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.619174 25589 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:56.619611  1239 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.620388  1216 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.620152  1239 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.625283  1239 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.619627 25589 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.628037  1254 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20240426 13:22:56.628540 25070 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.630138 25589 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:56.630578 25069 leader_election.cc:334] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.632586 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.635079  1216 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20240426 13:22:56.630934 25589 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240426 13:22:56.630183  1247 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.631997  1239 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.638602  1239 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.640630  1239 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.635591 25070 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.636714 25589 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.643234  1254 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.642378 24849 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 21 to 22, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 22 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:22:56.644307 25588 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.650133 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.650768 25622 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 29.
W20240426 13:22:56.654057 25069 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.654898 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.654433 25070 leader_election.cc:334] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.656643  1247 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:56.658033 25069 leader_election.cc:334] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.655297 25589 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.659158 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.662488 25070 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.660226  1247 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:56.663913 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.665200  1216 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20240426 13:22:56.665539  1216 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.665861  1216 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 29 FOLLOWER]: Advancing to term 30
W20240426 13:22:56.668354 25589 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.687572  1216 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.693343  1216 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:22:56.693953 25070 leader_election.cc:334] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.694250 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:22:56.694758 25622 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Advancing to term 30
I20240426 13:22:56.702903 25622 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 30.
I20240426 13:22:56.703823 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.708243  1216 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Leader election won for term 30
I20240426 13:22:56.709151  1216 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:22:56.710023  1216 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:56.724642 25589 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:56.726482 25589 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.734531 24849 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 29 to 30, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 30 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:22:56.747210 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:56.748754 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.749773 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.750876 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.751209  1216 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.751678  1216 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:22:56.754127 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.754967 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.756019 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.756592 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.757057 25069 leader_election.cc:334] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:56.757778 25070 leader_election.cc:334] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.758136 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.758833  1244 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:56.758955  1216 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.759385 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.761155 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.762290 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.765133 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.765952 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.767021 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.769874 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.770661 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.771369 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.774078 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.774868 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.775974 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.778702 23796 tablet_replica.cc:330] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.779541 23796 raft_consensus.cc:2229] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.780680 23796 raft_consensus.cc:2258] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.783450 23796 tablet_replica.cc:330] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.784291 23796 raft_consensus.cc:2229] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.784925 23796 raft_consensus.cc:2258] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.787537 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.788393 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.789238 23796 pending_rounds.cc:62] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Trying to abort 1 pending ops.
I20240426 13:22:56.789522 23796 pending_rounds.cc:68] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting op as it isn't in flight: id { term: 30 index: 20 } timestamp: 7021108333413793792 op_type: NO_OP noop_request { }
I20240426 13:22:56.790079 23796 raft_consensus.cc:2866] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:56.790421 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.793294 23796 tablet_replica.cc:330] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.794082 23796 raft_consensus.cc:2229] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.794670 23796 raft_consensus.cc:2258] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.797164 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.797927 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.798954 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.801525 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.802332 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.803375 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.805912 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.806730 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.807500 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.810153 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.810971 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.812197 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.814982 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.815802 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.815820 25589 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.816713  1216 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.817168 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.817236  1216 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.820158 23796 tablet_replica.cc:330] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.821007 23796 raft_consensus.cc:2229] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.822271 23796 raft_consensus.cc:2258] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.822652  1262 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.823210  1262 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:56.825038 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.825898 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.825960 25070 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.819113  1216 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.827217 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.827646 25069 leader_election.cc:334] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.828142 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.826036  1262 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:22:56.829689  1262 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20240426 13:22:56.829933 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.830907 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.831785 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.832178 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:22:56.832808 25622 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 23.
W20240426 13:22:56.833653 25070 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.834487 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:22:56.837244 25069 leader_election.cc:334] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.837716 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.838258  1262 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.835398 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.844473 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.846616 23796 tablet_replica.cc:330] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.847153 23796 raft_consensus.cc:2229] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.847976 23796 raft_consensus.cc:2258] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.849787 23796 tablet_replica.cc:330] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.850338 23796 raft_consensus.cc:2229] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.850812 23796 raft_consensus.cc:2258] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.853302 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.854558 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.855330 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.855660 25589 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:56.856420 25589 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.857533 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.858100 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.858857 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.860785 23796 tablet_replica.cc:330] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.861291 23796 raft_consensus.cc:2229] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.861771 23796 raft_consensus.cc:2258] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.863766 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.864322 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.864807 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.866614 23796 tablet_replica.cc:330] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.867153 23796 raft_consensus.cc:2229] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.867916 23796 raft_consensus.cc:2258] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.869634 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.870162 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.870774 23796 pending_rounds.cc:62] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Trying to abort 1 pending ops.
I20240426 13:22:56.870941 23796 pending_rounds.cc:68] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting op as it isn't in flight: id { term: 21 index: 18 } timestamp: 7021108332242690048 op_type: NO_OP noop_request { }
I20240426 13:22:56.871217 23796 raft_consensus.cc:2866] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:56.871462 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.873610 23796 tablet_replica.cc:330] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.874126 23796 raft_consensus.cc:2229] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.874585 23796 raft_consensus.cc:2258] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.876451 23796 tablet_replica.cc:330] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.877002 23796 raft_consensus.cc:2229] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.877727 23796 raft_consensus.cc:2258] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.880196 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.880861 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.881839 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.883878 23796 tablet_replica.cc:330] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.884479 23796 raft_consensus.cc:2229] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.885257 23796 raft_consensus.cc:2258] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.885825  1262 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.886353  1262 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.887558 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.888343 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.888523  1262 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.889385 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.892339 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.893031 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.892733 25069 leader_election.cc:334] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.893716 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 14 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.894861 25070 leader_election.cc:334] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.895372 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:56.896137  1262 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:56.896346 23796 tablet_replica.cc:330] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.898370 23796 raft_consensus.cc:2229] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.899389 23796 raft_consensus.cc:2258] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.901801 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.902473 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.902580  1262 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:56.903525 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.903216  1262 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:22:56.905759  1262 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:56.906302 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.908224 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.911845 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.914543 23796 tablet_replica.cc:330] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:22:56.913843 25069 leader_election.cc:334] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:56.915330 23796 raft_consensus.cc:2229] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:56.916013 25070 leader_election.cc:334] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:56.916271 23796 raft_consensus.cc:2258] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.916462 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:56.917060 25070 raft_consensus.cc:2670] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240426 13:22:56.918644 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.919226 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.919768 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.921821 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.922348 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.923206 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.925208 23796 tablet_replica.cc:330] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.925707 23796 raft_consensus.cc:2229] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.926244 23796 raft_consensus.cc:2258] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.928885 23796 tablet_replica.cc:330] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.929584 23796 raft_consensus.cc:2229] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.930142 23796 raft_consensus.cc:2258] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.931579 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:22:56.932729 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.933305 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.934007 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.936357 23796 tablet_replica.cc:330] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.936978 23796 raft_consensus.cc:2229] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:56.937836 23796 raft_consensus.cc:2258] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.939868 23796 tablet_replica.cc:330] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.940395 23796 raft_consensus.cc:2229] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.940932 23796 raft_consensus.cc:2258] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.942714 23796 tablet_replica.cc:330] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.943147 23796 raft_consensus.cc:2229] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.943704 23796 raft_consensus.cc:2258] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:56.945432 23796 tablet_replica.cc:330] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:22:56.945874 23796 raft_consensus.cc:2229] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:56.946270 23796 raft_consensus.cc:2258] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 21 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:56.956544 25588 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.009320 25589 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.011494 25588 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.100375 25589 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Couldn't send request to peer 30c2bb0adfa54bbfa3cb39773d4043c8. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.131366 25588 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.158466 25589 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:22:57.170512 25588 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.173583 25589 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.319000 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.324774 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.325039 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.330214 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.330530 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.334776 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.335296 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:57.343670 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.343924 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.370982 25589 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.441947 25589 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.481952 25589 consensus_peers.cc:485] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.489442 25588 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.508494 25589 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.509524 25588 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.527784 25588 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.626454 25589 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.640420 25589 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.667951 25589 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.682479 25588 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.703002 25588 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:57.708134 25588 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:57.796669 25589 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240426 13:22:57.803711  1258 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:57.804320  1258 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:57.806285  1258 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:57.810668 25589 leader_election.cc:334] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:57.811316 25589 leader_election.cc:334] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:57.813215 25589 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:57.813874  1254 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20240426 13:22:57.816398 25589 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:57.821712 25588 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240426 13:22:58.147639  1258 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.148284  1258 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.148903  1257 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.149438  1257 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.150650  1258 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.151350  1257 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.154484  1257 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.157277  1270 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.156983  1257 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.159183  1270 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.160490  1257 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.163852  1270 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:22:58.166326  1257 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.169032  1257 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.167595  1254 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.171701  1254 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.173908  1254 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.176744  1257 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.184625 25588 consensus_peers.cc:485] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:58.185174 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
W20240426 13:22:58.185348 25588 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240426 13:22:58.186256 25588 consensus_peers.cc:485] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240426 13:22:58.186908 25589 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240426 13:22:58.187532 25589 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.188019 25589 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.188745 25589 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.189304 25588 leader_election.cc:334] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.189767 25588 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.189996 25589 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.190153 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.190469 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.190943  1257 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20240426 13:22:58.191097 25588 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240426 13:22:58.191183  1254 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:58.191810 25588 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.192227 25589 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.192358  1254 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.192664 25589 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.192924  1254 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.193389  1272 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:58.193728 25589 leader_election.cc:334] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.194134 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.194911  1254 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.195014 25589 leader_election.cc:334] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.195513  1254 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20240426 13:22:58.195528 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.196524  1254 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20240426 13:22:58.198405 25588 leader_election.cc:334] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.200345 25589 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.201087 25589 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.201506 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.202234  1254 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20240426 13:22:58.204454 25589 leader_election.cc:334] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.204887 25589 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.205664  1254 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20240426 13:22:58.248847  1272 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.249392  1272 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.251585  1272 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.253271  1272 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.253767  1272 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.255791  1272 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:58.256992 25589 leader_election.cc:334] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.257680 25589 leader_election.cc:334] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.258119 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.258833  1272 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20240426 13:22:58.258967  1254 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:22:58.259649  1254 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:22:58.261215 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:58.261754  1254 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:22:58.261844 25589 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.263458 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.264086  1254 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20240426 13:22:58.265770 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
W20240426 13:22:58.268257 25589 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.268675 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.269403  1254 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20240426 13:22:58.271042 25588 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240426 13:22:58.283793 23796 tablet_server.cc:178] TabletServer@127.23.61.4:0 shutting down...
I20240426 13:22:58.317675 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:22:58.318392 23796 tablet_replica.cc:330] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.319039 23796 raft_consensus.cc:2229] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.319576 23796 raft_consensus.cc:2258] T f7408112bace482693c999f678aa9d73 P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.321659 23796 tablet_replica.cc:330] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.322213 23796 raft_consensus.cc:2229] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.322695 23796 raft_consensus.cc:2258] T 9108945c08ba4b2287c1e238b402722f P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.324673 23796 tablet_replica.cc:330] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.325186 23796 raft_consensus.cc:2229] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.325676 23796 raft_consensus.cc:2258] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.327610 23796 tablet_replica.cc:330] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.328176 23796 raft_consensus.cc:2229] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.328614 23796 raft_consensus.cc:2258] T 551154b9c3494e4da288affc4b88c361 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.330657 23796 tablet_replica.cc:330] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.331240 23796 raft_consensus.cc:2229] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.332337 23796 raft_consensus.cc:2258] T dacba3e2733347ecb1304d307310c944 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.333112  1254 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.333631  1254 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.335448 23796 tablet_replica.cc:330] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.335736  1254 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.336309 23796 raft_consensus.cc:2229] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.336846  1254 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.337317  1254 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.338933  1254 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:22:58.339082 25588 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Network error: Client connection negotiation failed: client connection to 127.23.61.1:41817: connect: Connection refused (error 111)
I20240426 13:22:58.339821 23796 raft_consensus.cc:2258] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.340462 25589 leader_election.cc:334] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.340806 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:22:58.341387  1272 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20240426 13:22:58.342209 23796 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.342834 23796 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus shutting down.
W20240426 13:22:58.343330 25589 leader_election.cc:334] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:58.343801 25589 leader_election.cc:334] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.343923 23796 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.344146 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:22:58.344630 25589 raft_consensus.cc:2670] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240426 13:22:58.346222 23796 tablet_replica.cc:330] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.347066 23796 raft_consensus.cc:2229] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.347680 23796 raft_consensus.cc:2258] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.350546 23796 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.351352 23796 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.352447 23796 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.355149 23796 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.355926 23796 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.356726 23796 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 30 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.359433 23796 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.360189 23796 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.361217 23796 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.363968 23796 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.364799 23796 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.365531 23796 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.368214 23796 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.368947 23796 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.369525 23796 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 26 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.372153 23796 tablet_replica.cc:330] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.372927 23796 raft_consensus.cc:2229] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.374099 23796 raft_consensus.cc:2258] T ece98ef0f07f4184bedf2cb96b628f20 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.376865 23796 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.377557 23796 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.378528 23796 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.381074 23796 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.381829 23796 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.382802 23796 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 15 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.385376 23796 tablet_replica.cc:330] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.386087 23796 raft_consensus.cc:2229] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.386551 23796 raft_consensus.cc:2258] T 0d8fda82f95348219809212d99d6704e P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.388743  1272 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:22:58.389185  1272 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:22:58.389492 23796 tablet_replica.cc:330] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.390343 23796 raft_consensus.cc:2229] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.391423 23796 raft_consensus.cc:2258] T f04bd8996e6349879e25fca4e52dbda9 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.390753  1272 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:22:58.394419 23796 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
W20240426 13:22:58.394776 25589 leader_election.cc:334] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.395150 23796 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.395741 23796 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.396235 25589 leader_election.cc:334] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.396564 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:22:58.397181  1272 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20240426 13:22:58.398433 23796 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.399178 23796 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.400120 23796 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 23 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.402710 23796 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.403416 23796 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.404373 23796 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 25 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.406716 23796 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.407385 23796 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.407979 23796 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.410519 23796 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.411149 23796 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.412117 23796 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.414573 23796 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.415148 23796 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.416069 23796 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.418408 23796 tablet_replica.cc:330] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.419036 23796 raft_consensus.cc:2229] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.419595 23796 raft_consensus.cc:2258] T 79bb183792fd4ae08dd4564ec82efc36 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.422169 23796 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.422847 23796 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.423807 23796 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.426101 23796 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.426728 23796 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.427193 23796 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.429136 23796 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.429617 23796 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Raft consensus shutting down.
W20240426 13:22:58.430234 24901 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.430481 23796 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.433560 23796 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.434307 23796 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.435200 23796 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 17 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.435815 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
W20240426 13:22:58.436089 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.3:38089: connect: Connection refused (error 111)
I20240426 13:22:58.437379 23796 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.437983 23796 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.438897 23796 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.439973 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.440261 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.441062 23796 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.441926 23796 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.442505 23796 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 27 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.444370 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
W20240426 13:22:58.444593 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.445053 23796 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.445633 23796 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.446589 23796 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 24 FOLLOWER]: Raft consensus is shut down!
W20240426 13:22:58.448781 24901 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.448868 23796 tablet_replica.cc:330] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
W20240426 13:22:58.449049 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.23.61.2:43671: connect: Connection refused (error 111)
I20240426 13:22:58.449515 23796 raft_consensus.cc:2229] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.450374 23796 raft_consensus.cc:2258] T aca59ad77ad642c48a7e7eed1635089d P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.452481 23796 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.452970 23796 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.453696 23796 pending_rounds.cc:62] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Trying to abort 1 pending ops.
I20240426 13:22:58.453860 23796 pending_rounds.cc:68] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting op as it isn't in flight: id { term: 22 index: 18 } timestamp: 7021108333025619968 op_type: NO_OP noop_request { }
I20240426 13:22:58.454125 23796 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240426 13:22:58.454375 23796 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 22 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.456096 23796 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.456586 23796 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.456985 23796 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.458719 23796 tablet_replica.cc:330] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.459223 23796 raft_consensus.cc:2229] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.460072 23796 raft_consensus.cc:2258] T 242110e36ca34176b9d522087d440de6 P 1590f80bdade41c6a755b8501c55fe16 [term 29 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.462093 23796 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.462554 23796 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.463263 23796 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.465066 23796 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.465602 23796 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.465996 23796 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 21 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.467787 23796 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.468291 23796 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.469009 23796 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 20 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.470695 23796 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: stopping tablet replica
I20240426 13:22:58.471109 23796 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus shutting down.
I20240426 13:22:58.471529 23796 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 19 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.748260 23796 tablet_server.cc:194] TabletServer@127.23.61.4:0 shutdown complete.
I20240426 13:22:58.804746 23796 master.cc:537] Master@127.23.61.62:34153 shutting down...
I20240426 13:22:58.828001 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:22:58.828554 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:22:58.828860 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: stopping tablet replica
W20240426 13:22:58.836789 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08000082e0 after lost signal to thread 23799
W20240426 13:22:58.837991 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080032f1c0 after lost signal to thread 24888
W20240426 13:22:58.839412 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080040c9c0 after lost signal to thread 25650
I20240426 13:22:59.341562 23796 master.cc:552] Master@127.23.61.62:34153 shutdown complete.
I20240426 13:22:59.379818 23796 test_util.cc:175] -----------------------------------------------
I20240426 13:22:59.380008 23796 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (262540 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20240426 13:22:59.382807 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:34177
I20240426 13:22:59.383814 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:22:59.389362  1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.389458  1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.389881  1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.392412 23796 server_base.cc:1034] running on GCE node
I20240426 13:22:59.395790 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:22:59.395974 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:22:59.396174 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137779396153 us; error 0 us; skew 500 ppm
I20240426 13:22:59.396677 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:22:59.402765 23796 webserver.cc:460] Webserver started at http://127.23.61.62:33239/ using document root <none> and password file <none>
I20240426 13:22:59.403254 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:22:59.403417 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:22:59.403689 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:22:59.404848 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "d342a68e43cc4fe09bbac496e74e2107"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.409077 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I20240426 13:22:59.412410  1284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.413167 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20240426 13:22:59.413458 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "d342a68e43cc4fe09bbac496e74e2107"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.413710 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:22:59.434252 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:22:59.435441 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:22:59.471951 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:34177
I20240426 13:22:59.472103  1335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:34177 every 8 connection(s)
I20240426 13:22:59.475865  1336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:22:59.485332  1336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107: Bootstrap starting.
I20240426 13:22:59.489835  1336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107: Neither blocks nor log segments found. Creating new log.
I20240426 13:22:59.493610  1336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107: No bootstrap required, opened a new log
I20240426 13:22:59.495417  1336 raft_consensus.cc:348] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER }
I20240426 13:22:59.495767  1336 raft_consensus.cc:374] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:22:59.495965  1336 raft_consensus.cc:729] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d342a68e43cc4fe09bbac496e74e2107, State: Initialized, Role: FOLLOWER
I20240426 13:22:59.496429  1336 consensus_queue.cc:260] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER }
I20240426 13:22:59.496832  1336 raft_consensus.cc:388] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:22:59.497021  1336 raft_consensus.cc:482] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:22:59.497217  1336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:22:59.501250  1336 raft_consensus.cc:504] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER }
I20240426 13:22:59.501684  1336 leader_election.cc:302] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d342a68e43cc4fe09bbac496e74e2107; no voters: 
I20240426 13:22:59.502737  1336 leader_election.cc:288] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 13:22:59.503038  1339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:22:59.504475  1339 raft_consensus.cc:686] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 1 LEADER]: Becoming Leader. State: Replica: d342a68e43cc4fe09bbac496e74e2107, State: Running, Role: LEADER
I20240426 13:22:59.505126  1339 consensus_queue.cc:237] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER }
I20240426 13:22:59.505599  1336 sys_catalog.cc:564] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:22:59.509567  1340 sys_catalog.cc:455] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d342a68e43cc4fe09bbac496e74e2107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER } }
I20240426 13:22:59.510833  1340 sys_catalog.cc:458] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [sys.catalog]: This master's current role is: LEADER
I20240426 13:22:59.510687  1341 sys_catalog.cc:455] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d342a68e43cc4fe09bbac496e74e2107. Latest consensus state: current_term: 1 leader_uuid: "d342a68e43cc4fe09bbac496e74e2107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d342a68e43cc4fe09bbac496e74e2107" member_type: VOTER } }
I20240426 13:22:59.511384  1341 sys_catalog.cc:458] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [sys.catalog]: This master's current role is: LEADER
I20240426 13:22:59.512573  1347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:22:59.518014  1347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:22:59.520768 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:22:59.525900  1347 catalog_manager.cc:1347] Generated new cluster ID: 7265b513c6414d18969263f53dbfadd8
I20240426 13:22:59.526165  1347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:22:59.546623  1347 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:22:59.547865  1347 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:22:59.558423  1347 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107: Generated new TSK 0
I20240426 13:22:59.558984  1347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:22:59.587635 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:22:59.593150  1357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.594978  1358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.596509  1360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.597352 23796 server_base.cc:1034] running on GCE node
I20240426 13:22:59.600373 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:22:59.600558 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:22:59.600705 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137779600689 us; error 0 us; skew 500 ppm
I20240426 13:22:59.601241 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:22:59.603610 23796 webserver.cc:460] Webserver started at http://127.23.61.1:46533/ using document root <none> and password file <none>
I20240426 13:22:59.604120 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:22:59.604288 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:22:59.604528 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:22:59.605616 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.609993 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240426 13:22:59.613126  1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.613859 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240426 13:22:59.614172 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.614545 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:22:59.642344 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:22:59.643465 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:22:59.644937 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:22:59.647085 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:22:59.647279 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.647502 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:22:59.647665 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.686162 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:43845
I20240426 13:22:59.686265  1427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:43845 every 8 connection(s)
I20240426 13:22:59.691272 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:22:59.697479  1432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.698539  1433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.702402 23796 server_base.cc:1034] running on GCE node
W20240426 13:22:59.703303  1435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.704129  1428 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34177
I20240426 13:22:59.704542  1428 heartbeater.cc:464] Registering TS with master...
I20240426 13:22:59.705181  1428 heartbeater.cc:511] Master 127.23.61.62:34177 requested a full tablet report, sending...
I20240426 13:22:59.705312 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:22:59.705570 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:22:59.705731 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137779705713 us; error 0 us; skew 500 ppm
I20240426 13:22:59.706344 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:22:59.707140  1301 ts_manager.cc:194] Registered new tserver with Master: 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:22:59.708874 23796 webserver.cc:460] Webserver started at http://127.23.61.2:35053/ using document root <none> and password file <none>
I20240426 13:22:59.709242  1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34504
I20240426 13:22:59.709368 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:22:59.709671 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:22:59.709944 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:22:59.711133 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "0d163e7cd1714dc48ebf662957edf6cf"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.715786 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240426 13:22:59.719210  1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.719942 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240426 13:22:59.720278 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "0d163e7cd1714dc48ebf662957edf6cf"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.720557 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:22:59.733759 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:22:59.734880 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:22:59.736363 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:22:59.738461 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:22:59.738649 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.738866 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:22:59.739014 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.776865 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:38131
I20240426 13:22:59.776968  1502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:38131 every 8 connection(s)
I20240426 13:22:59.781311 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:22:59.788713  1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.791882  1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.793596 23796 server_base.cc:1034] running on GCE node
W20240426 13:22:59.793635  1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.794054  1503 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34177
I20240426 13:22:59.794581  1503 heartbeater.cc:464] Registering TS with master...
I20240426 13:22:59.794930 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:22:59.795224 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:22:59.795449 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137779795427 us; error 0 us; skew 500 ppm
I20240426 13:22:59.795477  1503 heartbeater.cc:511] Master 127.23.61.62:34177 requested a full tablet report, sending...
I20240426 13:22:59.796414 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:22:59.797694  1301 ts_manager.cc:194] Registered new tserver with Master: 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:22:59.799031  1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34506
I20240426 13:22:59.799374 23796 webserver.cc:460] Webserver started at http://127.23.61.3:38131/ using document root <none> and password file <none>
I20240426 13:22:59.799875 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:22:59.800092 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:22:59.800344 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:22:59.801486 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "b156dff0c7b441c6b6f601006d0fcaea"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.805831 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240426 13:22:59.808871  1514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.809620 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20240426 13:22:59.809901 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "b156dff0c7b441c6b6f601006d0fcaea"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.810169 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:22:59.837881 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:22:59.839339 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:22:59.840790 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:22:59.843008 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:22:59.843216 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.843451 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:22:59.843598 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.883128 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:33613
I20240426 13:22:59.883217  1576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:33613 every 8 connection(s)
I20240426 13:22:59.887960 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:22:59.894536  1580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.899369  1581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:22:59.902647  1583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:22:59.902915  1577 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34177
I20240426 13:22:59.903612  1577 heartbeater.cc:464] Registering TS with master...
I20240426 13:22:59.904735  1577 heartbeater.cc:511] Master 127.23.61.62:34177 requested a full tablet report, sending...
I20240426 13:22:59.907498  1301 ts_manager.cc:194] Registered new tserver with Master: b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:22:59.908166 23796 server_base.cc:1034] running on GCE node
I20240426 13:22:59.908983  1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34512
I20240426 13:22:59.909081 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:22:59.909402 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:22:59.909579 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137779909560 us; error 0 us; skew 500 ppm
I20240426 13:22:59.910287 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:22:59.913039 23796 webserver.cc:460] Webserver started at http://127.23.61.4:39389/ using document root <none> and password file <none>
I20240426 13:22:59.913527 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:22:59.913730 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:22:59.913954 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:22:59.915148 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/instance:
uuid: "948b43c3ee7f4e209c7c91300198700c"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.919392 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240426 13:22:59.922550  1588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.923274 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240426 13:22:59.923554 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
uuid: "948b43c3ee7f4e209c7c91300198700c"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:22:59.923813 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:22:59.941413 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:22:59.942576 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:22:59.943962 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:22:59.946123 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:22:59.946301 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.946545 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:22:59.946702 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:22:59.986209 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.4:34539
I20240426 13:22:59.986305  1650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.4:34539 every 8 connection(s)
I20240426 13:22:59.998980  1651 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34177
I20240426 13:22:59.999372  1651 heartbeater.cc:464] Registering TS with master...
I20240426 13:23:00.000116  1651 heartbeater.cc:511] Master 127.23.61.62:34177 requested a full tablet report, sending...
I20240426 13:23:00.001847  1301 ts_manager.cc:194] Registered new tserver with Master: 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:00.002010 23796 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012984993s
I20240426 13:23:00.003692  1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34522
I20240426 13:23:00.004766 23796 test_util.cc:268] Using random seed: -847698040
I20240426 13:23:00.028369  1301 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34536:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240426 13:23:00.259223  1393 tablet_service.cc:1467] Processing CreateTablet for tablet b4c99cde390d4c2981806647821fd09b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:23:00.260913  1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c99cde390d4c2981806647821fd09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.270529  1542 tablet_service.cc:1467] Processing CreateTablet for tablet b4c99cde390d4c2981806647821fd09b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:23:00.271970  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c99cde390d4c2981806647821fd09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.272552  1392 tablet_service.cc:1467] Processing CreateTablet for tablet 3b813272c2184c4392e7f05b0011178f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:23:00.272564  1541 tablet_service.cc:1467] Processing CreateTablet for tablet 3b813272c2184c4392e7f05b0011178f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:23:00.274014  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b813272c2184c4392e7f05b0011178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.274214  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b813272c2184c4392e7f05b0011178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.270594  1468 tablet_service.cc:1467] Processing CreateTablet for tablet 3b813272c2184c4392e7f05b0011178f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:23:00.277050  1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b813272c2184c4392e7f05b0011178f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.278817  1540 tablet_service.cc:1467] Processing CreateTablet for tablet acd95475394f4c5bb64e94326a4cf9f5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:23:00.280203  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd95475394f4c5bb64e94326a4cf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.282189  1467 tablet_service.cc:1467] Processing CreateTablet for tablet 160c4150b51e426ab85b80343bc29efc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:23:00.283547  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160c4150b51e426ab85b80343bc29efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.295714  1466 tablet_service.cc:1467] Processing CreateTablet for tablet acd95475394f4c5bb64e94326a4cf9f5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:23:00.296535  1391 tablet_service.cc:1467] Processing CreateTablet for tablet 160c4150b51e426ab85b80343bc29efc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:23:00.297231  1672 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:00.297430  1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd95475394f4c5bb64e94326a4cf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.298031  1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160c4150b51e426ab85b80343bc29efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.301502  1539 tablet_service.cc:1467] Processing CreateTablet for tablet e186fc847c7549b58d5a94639edf09cf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:23:00.302096  1390 tablet_service.cc:1467] Processing CreateTablet for tablet e186fc847c7549b58d5a94639edf09cf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:23:00.302950  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186fc847c7549b58d5a94639edf09cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.303481  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186fc847c7549b58d5a94639edf09cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.306272  1465 tablet_service.cc:1467] Processing CreateTablet for tablet e186fc847c7549b58d5a94639edf09cf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:23:00.306386  1538 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc860ae90a4860bc539207a9259897 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:23:00.307758  1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e186fc847c7549b58d5a94639edf09cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.307857  1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc860ae90a4860bc539207a9259897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.311699  1464 tablet_service.cc:1467] Processing CreateTablet for tablet 0013296c2f6d4bec82b9277e5df43e64 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:23:00.313114  1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0013296c2f6d4bec82b9277e5df43e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.322660  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 0013296c2f6d4bec82b9277e5df43e64 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:23:00.325243  1388 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc860ae90a4860bc539207a9259897 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:23:00.325863  1616 tablet_service.cc:1467] Processing CreateTablet for tablet b4c99cde390d4c2981806647821fd09b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:23:00.327201  1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c99cde390d4c2981806647821fd09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.327697  1614 tablet_service.cc:1467] Processing CreateTablet for tablet acd95475394f4c5bb64e94326a4cf9f5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:23:00.329056  1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd95475394f4c5bb64e94326a4cf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.329202  1615 tablet_service.cc:1467] Processing CreateTablet for tablet 160c4150b51e426ab85b80343bc29efc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:23:00.330027  1672 tablet_bootstrap.cc:654] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.330539  1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160c4150b51e426ab85b80343bc29efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.333221  1613 tablet_service.cc:1467] Processing CreateTablet for tablet 0013296c2f6d4bec82b9277e5df43e64 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:23:00.334563  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0013296c2f6d4bec82b9277e5df43e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.336014  1612 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc860ae90a4860bc539207a9259897 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:23:00.345264  1463 tablet_service.cc:1467] Processing CreateTablet for tablet f376ec882d7545d8bb64b314b89dc70c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:23:00.346396  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0013296c2f6d4bec82b9277e5df43e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.347666  1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc860ae90a4860bc539207a9259897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.352502  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc860ae90a4860bc539207a9259897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.357290  1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f376ec882d7545d8bb64b314b89dc70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.372501  1542 tablet_service.cc:1467] Processing CreateTablet for tablet f376ec882d7545d8bb64b314b89dc70c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:23:00.373963  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f376ec882d7545d8bb64b314b89dc70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.382702  1611 tablet_service.cc:1467] Processing CreateTablet for tablet f376ec882d7545d8bb64b314b89dc70c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:23:00.383035  1392 tablet_service.cc:1467] Processing CreateTablet for tablet c2c76184e5b74fe4b94b24a896f5b89d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:23:00.384307  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f376ec882d7545d8bb64b314b89dc70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.384263  1540 tablet_service.cc:1467] Processing CreateTablet for tablet c2c76184e5b74fe4b94b24a896f5b89d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:23:00.385681  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c76184e5b74fe4b94b24a896f5b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.386237  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c76184e5b74fe4b94b24a896f5b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.386122  1462 tablet_service.cc:1467] Processing CreateTablet for tablet c2c76184e5b74fe4b94b24a896f5b89d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:23:00.387502  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c76184e5b74fe4b94b24a896f5b89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.389947  1610 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea2d370e87e4c90ad5c68256ea2b940 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:23:00.390489  1391 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea2d370e87e4c90ad5c68256ea2b940 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:23:00.391250  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea2d370e87e4c90ad5c68256ea2b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.391707  1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea2d370e87e4c90ad5c68256ea2b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.418816  1671 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:00.427834  1466 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea2d370e87e4c90ad5c68256ea2b940 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:23:00.429873  1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea2d370e87e4c90ad5c68256ea2b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.432222  1671 tablet_bootstrap.cc:654] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.453472  1539 tablet_service.cc:1467] Processing CreateTablet for tablet b698eb433b2d49c8b17447ce4c5b28e5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:23:00.455355  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b698eb433b2d49c8b17447ce4c5b28e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.483922  1673 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:00.490047  1674 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:00.490505  1613 tablet_service.cc:1467] Processing CreateTablet for tablet b698eb433b2d49c8b17447ce4c5b28e5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:23:00.491863  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b698eb433b2d49c8b17447ce4c5b28e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.494647  1465 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56eac9ef7e45e39b06bab0ddc3275b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:23:00.500986  1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56eac9ef7e45e39b06bab0ddc3275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.504151  1464 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb1fddf692c4843acc3c5f994fdf53e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:23:00.508257  1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb1fddf692c4843acc3c5f994fdf53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.509274  1542 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb1fddf692c4843acc3c5f994fdf53e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:23:00.512172  1540 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56eac9ef7e45e39b06bab0ddc3275b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:23:00.513509  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56eac9ef7e45e39b06bab0ddc3275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.513630  1390 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb1fddf692c4843acc3c5f994fdf53e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:23:00.514793  1467 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1fd3ed641c443095bc506c4c0107e7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:23:00.516057  1671 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:00.516301  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1fd3ed641c443095bc506c4c0107e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.516564  1671 ts_tablet_manager.cc:1367] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.098s	user 0.013s	sys 0.045s
W20240426 13:23:00.517014  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:00.518978  1541 tablet_service.cc:1467] Processing CreateTablet for tablet d5a787774e5341acaa26b09a1a61388f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:23:00.521728  1671 raft_consensus.cc:348] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:00.522889  1673 tablet_bootstrap.cc:654] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.520900  1388 tablet_service.cc:1467] Processing CreateTablet for tablet b698eb433b2d49c8b17447ce4c5b28e5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:23:00.539413  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1fd3ed641c443095bc506c4c0107e7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:23:00.516578  1616 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1fd3ed641c443095bc506c4c0107e7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:23:00.545694  1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1fd3ed641c443095bc506c4c0107e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.548138  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb1fddf692c4843acc3c5f994fdf53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.548902  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a787774e5341acaa26b09a1a61388f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.552139  1671 raft_consensus.cc:374] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:00.552525  1671 raft_consensus.cc:729] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:00.529605  1674 tablet_bootstrap.cc:654] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.557440  1538 tablet_service.cc:1467] Processing CreateTablet for tablet 074e277f90624d9e8754dfd0a7bb703d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:23:00.559052  1672 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:00.559574  1672 ts_tablet_manager.cc:1367] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.263s	user 0.015s	sys 0.057s
I20240426 13:23:00.562352  1463 tablet_service.cc:1467] Processing CreateTablet for tablet 074e277f90624d9e8754dfd0a7bb703d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:23:00.543910  1614 tablet_service.cc:1467] Processing CreateTablet for tablet 0c56eac9ef7e45e39b06bab0ddc3275b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:23:00.555801  1391 tablet_service.cc:1467] Processing CreateTablet for tablet d5a787774e5341acaa26b09a1a61388f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:23:00.565057  1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074e277f90624d9e8754dfd0a7bb703d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.565557  1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c56eac9ef7e45e39b06bab0ddc3275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.568259  1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074e277f90624d9e8754dfd0a7bb703d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.569662  1671 consensus_queue.cc:260] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:00.574029  1671 ts_tablet_manager.cc:1398] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.001s
I20240426 13:23:00.574574  1674 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:00.585939  1464 tablet_service.cc:1467] Processing CreateTablet for tablet 191d125f2e474efabc176055155755b0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:23:00.576423  1428 heartbeater.cc:503] Master 127.23.61.62:34177 was elected leader, sending a full tablet report...
I20240426 13:23:00.581013  1611 tablet_service.cc:1467] Processing CreateTablet for tablet 074e277f90624d9e8754dfd0a7bb703d (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:23:00.587414  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074e277f90624d9e8754dfd0a7bb703d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.589224  1671 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:00.575081  1612 tablet_service.cc:1467] Processing CreateTablet for tablet d5a787774e5341acaa26b09a1a61388f (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:23:00.584888  1393 tablet_service.cc:1467] Processing CreateTablet for tablet 191d125f2e474efabc176055155755b0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:23:00.586349  1674 ts_tablet_manager.cc:1367] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.097s	user 0.015s	sys 0.000s
I20240426 13:23:00.592718  1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a787774e5341acaa26b09a1a61388f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.595003  1674 raft_consensus.cc:348] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.595839  1674 raft_consensus.cc:374] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:00.596174  1674 raft_consensus.cc:729] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:00.582968  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb1fddf692c4843acc3c5f994fdf53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.597226  1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a787774e5341acaa26b09a1a61388f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.601333  1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d125f2e474efabc176055155755b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.602007  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1fd3ed641c443095bc506c4c0107e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.604478  1673 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:00.605031  1673 ts_tablet_manager.cc:1367] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.121s	user 0.027s	sys 0.027s
I20240426 13:23:00.606320  1539 tablet_service.cc:1467] Processing CreateTablet for tablet 191d125f2e474efabc176055155755b0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:23:00.607721  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d125f2e474efabc176055155755b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.612017  1674 consensus_queue.cc:260] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.607893  1673 raft_consensus.cc:348] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:00.614151  1673 raft_consensus.cc:374] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:00.614440  1673 raft_consensus.cc:729] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:00.614552  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b698eb433b2d49c8b17447ce4c5b28e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.617929  1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191d125f2e474efabc176055155755b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.621222  1611 tablet_service.cc:1467] Processing CreateTablet for tablet 7b84898ee62e4f8eb0fb983c9c927fe0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:23:00.626418  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b84898ee62e4f8eb0fb983c9c927fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.615259  1673 consensus_queue.cc:260] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:00.638878  1671 tablet_bootstrap.cc:654] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.642508  1392 tablet_service.cc:1467] Processing CreateTablet for tablet 7b84898ee62e4f8eb0fb983c9c927fe0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:23:00.662994  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b84898ee62e4f8eb0fb983c9c927fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.663064  1462 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb623ed697340f08d33b2d5e98eaedf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:23:00.644904  1466 tablet_service.cc:1467] Processing CreateTablet for tablet 7b84898ee62e4f8eb0fb983c9c927fe0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:23:00.646554  1615 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccda8d3014c4397a7e8987acdb33799 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:23:00.647922  1387 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccda8d3014c4397a7e8987acdb33799 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:23:00.638921  1672 raft_consensus.cc:348] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.652019  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb623ed697340f08d33b2d5e98eaedf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:23:00.652112  1610 tablet_service.cc:1467] Processing CreateTablet for tablet eb745ab6644f4cfb9c10676dadd72f29 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:23:00.653239  1467 tablet_service.cc:1467] Processing CreateTablet for tablet eb745ab6644f4cfb9c10676dadd72f29 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:23:00.653836  1537 tablet_service.cc:1467] Processing CreateTablet for tablet eb745ab6644f4cfb9c10676dadd72f29 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:23:00.669350  1651 heartbeater.cc:503] Master 127.23.61.62:34177 was elected leader, sending a full tablet report...
I20240426 13:23:00.670040  1674 ts_tablet_manager.cc:1398] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.079s	user 0.010s	sys 0.003s
I20240426 13:23:00.669948  1681 raft_consensus.cc:482] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:00.682636  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb745ab6644f4cfb9c10676dadd72f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.682916  1681 raft_consensus.cc:504] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.684566  1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccda8d3014c4397a7e8987acdb33799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.685454  1681 leader_election.cc:288] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:00.669582  1538 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccda8d3014c4397a7e8987acdb33799 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:23:00.675264  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb623ed697340f08d33b2d5e98eaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.672736  1672 raft_consensus.cc:374] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 13:23:00.691020  1429 tablet.cc:2369] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 13:23:00.694512  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb745ab6644f4cfb9c10676dadd72f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.694263  1391 tablet_service.cc:1467] Processing CreateTablet for tablet 442e14b352624bdbb04e9a3fdc445d32 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:23:00.682693  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb623ed697340f08d33b2d5e98eaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.702903  1536 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb623ed697340f08d33b2d5e98eaedf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:23:00.703286  1468 tablet_service.cc:1467] Processing CreateTablet for tablet 442e14b352624bdbb04e9a3fdc445d32 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:23:00.706004  1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccda8d3014c4397a7e8987acdb33799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.707865  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb745ab6644f4cfb9c10676dadd72f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.708506  1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442e14b352624bdbb04e9a3fdc445d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.710367  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccda8d3014c4397a7e8987acdb33799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.714217  1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b84898ee62e4f8eb0fb983c9c927fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.721087  1673 ts_tablet_manager.cc:1398] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.116s	user 0.012s	sys 0.021s
I20240426 13:23:00.721846  1542 tablet_service.cc:1467] Processing CreateTablet for tablet 8f73751c65ca4d568467edaf86391737 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:23:00.688616  1672 raft_consensus.cc:729] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:00.735469  1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442e14b352624bdbb04e9a3fdc445d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.744400  1673 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:00.748227  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb623ed697340f08d33b2d5e98eaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.764516  1503 heartbeater.cc:503] Master 127.23.61.62:34177 was elected leader, sending a full tablet report...
I20240426 13:23:00.764909  1673 tablet_bootstrap.cc:654] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.770979  1672 consensus_queue.cc:260] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.776160  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f73751c65ca4d568467edaf86391737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.785845  1674 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:00.789619  1540 tablet_service.cc:1467] Processing CreateTablet for tablet 9cadbce7a42d4cc181613ffd7ae24190 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:23:00.791002  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cadbce7a42d4cc181613ffd7ae24190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.797048  1465 tablet_service.cc:1467] Processing CreateTablet for tablet 9cadbce7a42d4cc181613ffd7ae24190 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:23:00.798883  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 8f73751c65ca4d568467edaf86391737 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:23:00.800201  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f73751c65ca4d568467edaf86391737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.820608  1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cadbce7a42d4cc181613ffd7ae24190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.824728  1674 tablet_bootstrap.cc:654] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:00.862771  1671 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:00.863454  1671 ts_tablet_manager.cc:1367] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.275s	user 0.007s	sys 0.010s
I20240426 13:23:00.882131  1577 heartbeater.cc:503] Master 127.23.61.62:34177 was elected leader, sending a full tablet report...
I20240426 13:23:00.888075  1464 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7cffa351e841f2bbf2789cc31939cc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:23:00.888617  1538 tablet_service.cc:1467] Processing CreateTablet for tablet 612eff866f5b4a9e902f703f3f11eb82 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:23:00.892261  1672 ts_tablet_manager.cc:1398] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.332s	user 0.015s	sys 0.009s
I20240426 13:23:00.892271  1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612eff866f5b4a9e902f703f3f11eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.893584  1672 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:00.894413  1671 raft_consensus.cc:348] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.895198  1671 raft_consensus.cc:374] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:00.895469  1671 raft_consensus.cc:729] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:00.889428  1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7cffa351e841f2bbf2789cc31939cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.896486  1387 tablet_service.cc:1467] Processing CreateTablet for tablet 612eff866f5b4a9e902f703f3f11eb82 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:23:00.901822  1537 tablet_service.cc:1467] Processing CreateTablet for tablet 33617d7b11304e939632cf23bc38f168 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:23:00.903263  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33617d7b11304e939632cf23bc38f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.907568  1539 tablet_service.cc:1467] Processing CreateTablet for tablet d406b2568d574aedb189a69254c662aa (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:23:00.908865  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d406b2568d574aedb189a69254c662aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.909011  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7cffa351e841f2bbf2789cc31939cc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:23:00.897442  1682 raft_consensus.cc:482] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:00.910043  1682 raft_consensus.cc:504] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:00.911528  1541 tablet_service.cc:1467] Processing CreateTablet for tablet 919c4656a25c42e6a02d8d0a8d1f7f59 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:23:00.912828  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c4656a25c42e6a02d8d0a8d1f7f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.914026  1388 tablet_service.cc:1467] Processing CreateTablet for tablet 33617d7b11304e939632cf23bc38f168 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:23:00.901736  1463 tablet_service.cc:1467] Processing CreateTablet for tablet 612eff866f5b4a9e902f703f3f11eb82 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:23:00.916160  1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612eff866f5b4a9e902f703f3f11eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.916386  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33617d7b11304e939632cf23bc38f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.918648  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7cffa351e841f2bbf2789cc31939cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.920465  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612eff866f5b4a9e902f703f3f11eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.932227  1671 consensus_queue.cc:260] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.938508  1390 tablet_service.cc:1467] Processing CreateTablet for tablet 919c4656a25c42e6a02d8d0a8d1f7f59 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:23:00.939843  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c4656a25c42e6a02d8d0a8d1f7f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.940986  1467 tablet_service.cc:1467] Processing CreateTablet for tablet 919c4656a25c42e6a02d8d0a8d1f7f59 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:23:00.941067  1611 tablet_service.cc:1467] Processing CreateTablet for tablet 442e14b352624bdbb04e9a3fdc445d32 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:23:00.943789  1615 tablet_service.cc:1467] Processing CreateTablet for tablet 9cadbce7a42d4cc181613ffd7ae24190 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:23:00.945277  1610 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7cffa351e841f2bbf2789cc31939cc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:23:00.958665  1673 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:00.962116  1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cadbce7a42d4cc181613ffd7ae24190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.963670  1616 tablet_service.cc:1467] Processing CreateTablet for tablet 33617d7b11304e939632cf23bc38f168 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:23:00.965011  1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33617d7b11304e939632cf23bc38f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.968327  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7cffa351e841f2bbf2789cc31939cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.970604  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442e14b352624bdbb04e9a3fdc445d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.973409  1612 tablet_service.cc:1467] Processing CreateTablet for tablet 8f73751c65ca4d568467edaf86391737 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:23:00.974690  1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f73751c65ca4d568467edaf86391737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.952827  1682 leader_election.cc:288] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:00.978682  1671 ts_tablet_manager.cc:1398] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.115s	user 0.000s	sys 0.009s
I20240426 13:23:00.978717  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:00.979682  1552 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:00.981246  1590 leader_election.cc:302] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:00.982321  1680 raft_consensus.cc:2781] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:00.982695  1680 raft_consensus.cc:482] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:00.982996  1680 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:00.989683  1680 raft_consensus.cc:504] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:00.993325  1671 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:00.994138  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919c4656a25c42e6a02d8d0a8d1f7f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:00.993969  1468 tablet_service.cc:1467] Processing CreateTablet for tablet 95477b67a3084e64a196d9d64957a3f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:23:00.995620  1614 tablet_service.cc:1467] Processing CreateTablet for tablet d406b2568d574aedb189a69254c662aa (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:23:00.996769  1462 tablet_service.cc:1467] Processing CreateTablet for tablet d406b2568d574aedb189a69254c662aa (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:23:00.996857  1392 tablet_service.cc:1467] Processing CreateTablet for tablet 95477b67a3084e64a196d9d64957a3f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:23:00.997123  1542 tablet_service.cc:1467] Processing CreateTablet for tablet 289eab6dc655403aa7b8a801972d0eaf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:23:00.998363  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:00.998557  1613 tablet_service.cc:1467] Processing CreateTablet for tablet 95477b67a3084e64a196d9d64957a3f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:23:01.004712  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289eab6dc655403aa7b8a801972d0eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.005179  1609 tablet_service.cc:1467] Processing CreateTablet for tablet 134a02c7dd614ad8b152d809ebd56155 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:23:01.006464  1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a02c7dd614ad8b152d809ebd56155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.006837  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 134a02c7dd614ad8b152d809ebd56155 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:23:01.008888  1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d406b2568d574aedb189a69254c662aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.010064  1461 tablet_service.cc:1467] Processing CreateTablet for tablet 289eab6dc655403aa7b8a801972d0eaf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:23:01.010692  1403 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:01.011463  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95477b67a3084e64a196d9d64957a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.000736  1673 ts_tablet_manager.cc:1367] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.257s	user 0.015s	sys 0.011s
I20240426 13:23:01.018051  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95477b67a3084e64a196d9d64957a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.024353  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a02c7dd614ad8b152d809ebd56155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.025264  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:01.026763  1441 leader_election.cc:341] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.027894  1536 tablet_service.cc:1467] Processing CreateTablet for tablet 134a02c7dd614ad8b152d809ebd56155 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:23:01.029208  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a02c7dd614ad8b152d809ebd56155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.031342  1679 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:01.031831  1679 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.034376  1679 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:01.034922  1680 leader_election.cc:288] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:01.034279  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:01.036315  1552 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.036798  1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289eab6dc655403aa7b8a801972d0eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.040591  1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95477b67a3084e64a196d9d64957a3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.056358  1672 tablet_bootstrap.cc:654] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.061560  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d406b2568d574aedb189a69254c662aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.062944  1674 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:01.063335  1460 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc8321cda547cc94f2dcdd71507e28 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:23:01.068936  1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc8321cda547cc94f2dcdd71507e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.066256  1608 tablet_service.cc:1467] Processing CreateTablet for tablet 289eab6dc655403aa7b8a801972d0eaf (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:23:01.072539  1552 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:01.072957  1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289eab6dc655403aa7b8a801972d0eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.073740  1590 leader_election.cc:302] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:01.074530  1671 tablet_bootstrap.cc:654] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.091800  1537 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc8321cda547cc94f2dcdd71507e28 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:23:01.094199  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc8321cda547cc94f2dcdd71507e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.086817  1674 ts_tablet_manager.cc:1367] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.301s	user 0.008s	sys 0.009s
I20240426 13:23:01.083916  1680 raft_consensus.cc:2781] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:01.106272  1680 raft_consensus.cc:686] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:01.111979  1680 consensus_queue.cc:237] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.116205  1673 raft_consensus.cc:348] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.122231  1673 raft_consensus.cc:374] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.129597  1673 raft_consensus.cc:729] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.130081  1607 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3bd4a614b14384a7d4945594ab3871 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:23:01.131899  1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3bd4a614b14384a7d4945594ab3871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.136297  1673 consensus_queue.cc:260] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.141712  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:01.143944  1443 leader_election.cc:341] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.144564  1443 leader_election.cc:302] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:01.171473  1682 raft_consensus.cc:2726] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:01.180003  1674 raft_consensus.cc:348] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.180866  1674 raft_consensus.cc:374] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.181177  1674 raft_consensus.cc:729] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.181933  1674 consensus_queue.cc:260] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.200222  1673 ts_tablet_manager.cc:1398] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.158s	user 0.010s	sys 0.000s
I20240426 13:23:01.201296  1673 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:01.207834  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:01.208528  1403 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.194388  1674 ts_tablet_manager.cc:1398] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.087s	user 0.009s	sys 0.000s
I20240426 13:23:01.210260  1462 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3bd4a614b14384a7d4945594ab3871 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:23:01.211328  1674 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:01.216336  1672 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:01.216907  1672 ts_tablet_manager.cc:1367] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.324s	user 0.019s	sys 0.011s
I20240426 13:23:01.216814  1541 tablet_service.cc:1467] Processing CreateTablet for tablet a825e64679024315981ae5d0de008472 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:23:01.218267  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a825e64679024315981ae5d0de008472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.222782  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:01.219980  1672 raft_consensus.cc:348] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.223753  1672 raft_consensus.cc:374] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.224094  1672 raft_consensus.cc:729] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.221948  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc8321cda547cc94f2dcdd71507e28 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:23:01.225229  1673 tablet_bootstrap.cc:654] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.225651  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc8321cda547cc94f2dcdd71507e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.226802  1403 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:01.228672  1674 tablet_bootstrap.cc:654] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.222004  1613 tablet_service.cc:1467] Processing CreateTablet for tablet a825e64679024315981ae5d0de008472 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:23:01.229609  1298 catalog_manager.cc:5484] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:01.230337  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a825e64679024315981ae5d0de008472. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:23:01.234575  1367 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.235356  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3bd4a614b14384a7d4945594ab3871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.232427  1672 consensus_queue.cc:260] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.245203  1536 tablet_service.cc:1467] Processing CreateTablet for tablet 51698a9fac6a431499553247279e5250 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:23:01.246515  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51698a9fac6a431499553247279e5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.246917  1610 tablet_service.cc:1467] Processing CreateTablet for tablet 51698a9fac6a431499553247279e5250 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:23:01.247279  1392 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3bd4a614b14384a7d4945594ab3871 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:23:01.248631  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51698a9fac6a431499553247279e5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.253715  1387 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce3aa126ef422ca8fedc1e1349087b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:23:01.254460  1390 tablet_service.cc:1467] Processing CreateTablet for tablet a825e64679024315981ae5d0de008472 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:23:01.261083  1467 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce3aa126ef422ca8fedc1e1349087b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:23:01.248657  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3bd4a614b14384a7d4945594ab3871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.265230  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:01.271953  1478 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:01.272130  1671 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:01.274080  1671 ts_tablet_manager.cc:1367] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.281s	user 0.019s	sys 0.015s
I20240426 13:23:01.274293  1366 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:01.275038  1542 tablet_service.cc:1467] Processing CreateTablet for tablet a8ce3aa126ef422ca8fedc1e1349087b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:23:01.275529  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce3aa126ef422ca8fedc1e1349087b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.276209  1679 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:01.276646  1679 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:01.265836  1672 ts_tablet_manager.cc:1398] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.048s	user 0.004s	sys 0.004s
I20240426 13:23:01.276997  1679 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.277839  1672 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:01.280472  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce3aa126ef422ca8fedc1e1349087b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.272694  1468 tablet_service.cc:1467] Processing CreateTablet for tablet 51698a9fac6a431499553247279e5250 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:23:01.281574  1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51698a9fac6a431499553247279e5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.283879  1674 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:01.284402  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ce3aa126ef422ca8fedc1e1349087b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.285560  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a825e64679024315981ae5d0de008472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.289461  1679 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.292299  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:01.292940  1478 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.293445  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:01.294852  1673 tablet_bootstrap.cc:492] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
W20240426 13:23:01.294983  1367 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.299647  1478 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:01.297111  1671 raft_consensus.cc:348] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.304971  1366 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:01.292407  1679 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:01.284472  1674 ts_tablet_manager.cc:1367] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.073s	user 0.011s	sys 0.001s
I20240426 13:23:01.307073  1671 raft_consensus.cc:374] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.320473  1671 raft_consensus.cc:729] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.321540  1671 consensus_queue.cc:260] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.325614  1672 tablet_bootstrap.cc:654] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.328284  1679 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:01.328994  1679 raft_consensus.cc:686] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:01.330099  1679 consensus_queue.cc:237] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.295383  1673 ts_tablet_manager.cc:1367] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.094s	user 0.007s	sys 0.025s
I20240426 13:23:01.336881  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 7e00f8df835646e89148ac3e3a2d6abc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:23:01.338225  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e00f8df835646e89148ac3e3a2d6abc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:23:01.351248  1366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 46383.3us
I20240426 13:23:01.353947  1461 tablet_service.cc:1467] Processing CreateTablet for tablet 7e00f8df835646e89148ac3e3a2d6abc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:23:01.309963  1609 tablet_service.cc:1467] Processing CreateTablet for tablet 7e00f8df835646e89148ac3e3a2d6abc (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:23:01.356258  1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e00f8df835646e89148ac3e3a2d6abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.364380  1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e00f8df835646e89148ac3e3a2d6abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.349808  1673 raft_consensus.cc:348] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.371158  1673 raft_consensus.cc:374] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.370947  1542 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b9b0be82342918f3a75e964df8da7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:23:01.371549  1673 raft_consensus.cc:729] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.370385  1387 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b9b0be82342918f3a75e964df8da7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:23:01.372830  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b9b0be82342918f3a75e964df8da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.372606  1673 consensus_queue.cc:260] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.375715  1673 ts_tablet_manager.cc:1398] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.039s	user 0.003s	sys 0.003s
I20240426 13:23:01.376955  1673 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:01.378422  1463 tablet_service.cc:1467] Processing CreateTablet for tablet 8276c0795c4c4dab902ce8acda30340b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:23:01.378875  1537 tablet_service.cc:1467] Processing CreateTablet for tablet 8276c0795c4c4dab902ce8acda30340b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:23:01.379665  1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8276c0795c4c4dab902ce8acda30340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.380096  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8276c0795c4c4dab902ce8acda30340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.381611  1671 ts_tablet_manager.cc:1398] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.107s	user 0.014s	sys 0.006s
I20240426 13:23:01.383033  1671 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:01.387710  1607 tablet_service.cc:1467] Processing CreateTablet for tablet 8276c0795c4c4dab902ce8acda30340b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:23:01.388484  1536 tablet_service.cc:1467] Processing CreateTablet for tablet df9a043f048f4bf4acb06f582821197b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:23:01.389000  1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8276c0795c4c4dab902ce8acda30340b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.389688  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9a043f048f4bf4acb06f582821197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.394551  1460 tablet_service.cc:1467] Processing CreateTablet for tablet df9a043f048f4bf4acb06f582821197b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:23:01.394992  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b9b0be82342918f3a75e964df8da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.394860  1388 tablet_service.cc:1467] Processing CreateTablet for tablet df9a043f048f4bf4acb06f582821197b (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:23:01.395809  1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9a043f048f4bf4acb06f582821197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.396106  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9a043f048f4bf4acb06f582821197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.402720  1674 raft_consensus.cc:348] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.403525  1674 raft_consensus.cc:374] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.403815  1674 raft_consensus.cc:729] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.404558  1674 consensus_queue.cc:260] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.410118  1610 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b9b0be82342918f3a75e964df8da7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:23:01.410974  1673 tablet_bootstrap.cc:654] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.414732  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b9b0be82342918f3a75e964df8da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.428351  1671 tablet_bootstrap.cc:654] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.433331  1674 ts_tablet_manager.cc:1398] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.101s	user 0.002s	sys 0.004s
I20240426 13:23:01.434173  1674 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:01.445689  1608 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2388dc8c06400baf4b699cfa351544 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:23:01.446995  1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2388dc8c06400baf4b699cfa351544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.453795  1301 catalog_manager.cc:5484] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:01.457952  1674 tablet_bootstrap.cc:654] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.458256  1392 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2388dc8c06400baf4b699cfa351544 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:23:01.462153  1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2388dc8c06400baf4b699cfa351544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.470799  1390 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0863dde30e4fdab1de8f22d75fabd5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:23:01.477818  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0863dde30e4fdab1de8f22d75fabd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.470924  1613 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0863dde30e4fdab1de8f22d75fabd5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:23:01.482806  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0863dde30e4fdab1de8f22d75fabd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.483923  1541 tablet_service.cc:1467] Processing CreateTablet for tablet dc7a62fac3114417ae31d5ce8f6bbd86 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:23:01.485164  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7a62fac3114417ae31d5ce8f6bbd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.472097  1462 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2388dc8c06400baf4b699cfa351544 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:23:01.489537  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2388dc8c06400baf4b699cfa351544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.489374  1611 tablet_service.cc:1467] Processing CreateTablet for tablet dc7a62fac3114417ae31d5ce8f6bbd86 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:23:01.490674  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7a62fac3114417ae31d5ce8f6bbd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.492950  1539 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe65507a824cd88e8376761b8ebe1c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:23:01.493392  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe65507a824cd88e8376761b8ebe1c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:23:01.494204  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe65507a824cd88e8376761b8ebe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.494594  1672 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:01.494622  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe65507a824cd88e8376761b8ebe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.495800  1467 tablet_service.cc:1467] Processing CreateTablet for tablet dc7a62fac3114417ae31d5ce8f6bbd86 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:23:01.496150  1612 tablet_service.cc:1467] Processing CreateTablet for tablet f5eec2b6b55d4310bfd18eb2f7b093a5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:23:01.497591  1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eec2b6b55d4310bfd18eb2f7b093a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.498281  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7a62fac3114417ae31d5ce8f6bbd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.509748  1461 tablet_service.cc:1467] Processing CreateTablet for tablet 4efe65507a824cd88e8376761b8ebe1c (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:23:01.511183  1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4efe65507a824cd88e8376761b8ebe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.514978  1537 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0863dde30e4fdab1de8f22d75fabd5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:23:01.516451  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0863dde30e4fdab1de8f22d75fabd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.523646  1680 raft_consensus.cc:482] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:01.495132  1672 ts_tablet_manager.cc:1367] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.217s	user 0.030s	sys 0.046s
I20240426 13:23:01.524636  1686 raft_consensus.cc:482] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:01.524205  1680 raft_consensus.cc:504] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.525151  1686 raft_consensus.cc:504] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.526921  1680 leader_election.cc:288] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:01.527460  1686 leader_election.cc:288] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:01.530756  1540 tablet_service.cc:1467] Processing CreateTablet for tablet 10594296adca430d8519c32089193198 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:23:01.532159  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10594296adca430d8519c32089193198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.532517  1460 tablet_service.cc:1467] Processing CreateTablet for tablet f5eec2b6b55d4310bfd18eb2f7b093a5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:23:01.533821  1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eec2b6b55d4310bfd18eb2f7b093a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:23:01.536739  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:01.536475  1387 tablet_service.cc:1467] Processing CreateTablet for tablet f5eec2b6b55d4310bfd18eb2f7b093a5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:23:01.527637  1672 raft_consensus.cc:348] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.578336  1609 tablet_service.cc:1467] Processing CreateTablet for tablet 10594296adca430d8519c32089193198 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:23:01.579389  1672 raft_consensus.cc:374] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.579881  1672 raft_consensus.cc:729] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.580787  1672 consensus_queue.cc:260] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.582995  1672 ts_tablet_manager.cc:1398] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.058s	user 0.006s	sys 0.000s
I20240426 13:23:01.588979  1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10594296adca430d8519c32089193198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.594246  1679 raft_consensus.cc:482] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:01.595000  1679 raft_consensus.cc:504] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.583940  1672 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:01.598929  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:01.607396  1366 leader_election.cc:341] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.601079  1679 leader_election.cc:288] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:01.620182  1672 tablet_bootstrap.cc:654] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.620870  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eec2b6b55d4310bfd18eb2f7b093a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.626853  1541 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2512b5c104b73a15c3238bfedbe80 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:23:01.629788  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2512b5c104b73a15c3238bfedbe80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.630896  1607 tablet_service.cc:1467] Processing CreateTablet for tablet 54bf83b19a6047a7bec69cda298d5436 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:23:01.632180  1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bf83b19a6047a7bec69cda298d5436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.632993  1389 tablet_service.cc:1467] Processing CreateTablet for tablet 9766d6635658449298fdd0a3b0edae26 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:23:01.634223  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9766d6635658449298fdd0a3b0edae26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.636930  1542 tablet_service.cc:1467] Processing CreateTablet for tablet 9766d6635658449298fdd0a3b0edae26 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:23:01.638170  1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9766d6635658449298fdd0a3b0edae26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.638346  1388 tablet_service.cc:1467] Processing CreateTablet for tablet 10594296adca430d8519c32089193198 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:23:01.639971  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10594296adca430d8519c32089193198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.662564  1671 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:01.669955  1671 ts_tablet_manager.cc:1367] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.287s	user 0.018s	sys 0.027s
I20240426 13:23:01.685513  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:01.694589  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 54bf83b19a6047a7bec69cda298d5436 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:23:01.694928  1552 raft_consensus.cc:2454] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:01.700462  1671 raft_consensus.cc:348] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.701242  1671 raft_consensus.cc:374] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.701524  1671 raft_consensus.cc:729] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.702267  1671 consensus_queue.cc:260] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.707212  1608 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca73bd7cd06413fbad853c7ccc749f2 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:23:01.707754  1541 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca73bd7cd06413fbad853c7ccc749f2 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:23:01.708565  1608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca73bd7cd06413fbad853c7ccc749f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.725541  1463 tablet_service.cc:1467] Processing CreateTablet for tablet 54bf83b19a6047a7bec69cda298d5436 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:23:01.726817  1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bf83b19a6047a7bec69cda298d5436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.718271  1611 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2512b5c104b73a15c3238bfedbe80 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:23:01.730773  1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2512b5c104b73a15c3238bfedbe80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.718791  1391 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca73bd7cd06413fbad853c7ccc749f2 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:23:01.720484  1673 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:01.737780  1673 ts_tablet_manager.cc:1367] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.361s	user 0.062s	sys 0.064s
I20240426 13:23:01.737977  1671 ts_tablet_manager.cc:1398] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.054s	user 0.007s	sys 0.000s
I20240426 13:23:01.739081  1610 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae5495d004b4fc0a78ef115fb4642f1 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:23:01.740339  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae5495d004b4fc0a78ef115fb4642f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.741092  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:01.741765  1478 raft_consensus.cc:2454] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:01.741948  1467 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc2512b5c104b73a15c3238bfedbe80 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:23:01.743755  1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc2512b5c104b73a15c3238bfedbe80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.745079  1468 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae5495d004b4fc0a78ef115fb4642f1 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:23:01.718508  1536 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae5495d004b4fc0a78ef115fb4642f1 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:23:01.717727  1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca73bd7cd06413fbad853c7ccc749f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.753829  1515 leader_election.cc:302] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:01.720181  1590 leader_election.cc:302] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:01.754766  1686 raft_consensus.cc:2781] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:01.755313  1686 raft_consensus.cc:482] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:01.755738  1686 raft_consensus.cc:3037] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.757078  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54bf83b19a6047a7bec69cda298d5436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.760219  1680 raft_consensus.cc:2781] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:01.760640  1680 raft_consensus.cc:482] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:01.760975  1680 raft_consensus.cc:3037] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.720806  1461 tablet_service.cc:1467] Processing CreateTablet for tablet 9766d6635658449298fdd0a3b0edae26 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:23:01.767866  1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae5495d004b4fc0a78ef115fb4642f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.768376  1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca73bd7cd06413fbad853c7ccc749f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.777674  1718 consensus_queue.cc:1035] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:01.778246  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:01.779677  1368 leader_election.cc:341] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:01.780160  1368 leader_election.cc:302] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:01.781296  1718 raft_consensus.cc:482] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:01.782049  1679 raft_consensus.cc:2726] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:01.781800  1718 raft_consensus.cc:504] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.787264  1680 raft_consensus.cc:504] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.775213  1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9766d6635658449298fdd0a3b0edae26. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:23:01.788964  1285 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 46050.1us
I20240426 13:23:01.740576  1673 raft_consensus.cc:348] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.792433  1673 raft_consensus.cc:374] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.792527  1671 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:01.792788  1673 raft_consensus.cc:729] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.793566  1673 consensus_queue.cc:260] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:01.794843  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae5495d004b4fc0a78ef115fb4642f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.795797  1673 ts_tablet_manager.cc:1398] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.058s	user 0.005s	sys 0.000s
I20240426 13:23:01.749544  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:01.801026  1462 tablet_service.cc:1467] Processing CreateTablet for tablet f6b50872bb2142fe8a7a0c2e234d24af (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:23:01.806414  1539 tablet_service.cc:1467] Processing CreateTablet for tablet f6b50872bb2142fe8a7a0c2e234d24af (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:23:01.807691  1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b50872bb2142fe8a7a0c2e234d24af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.814846  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:01.819149  1390 tablet_service.cc:1467] Processing CreateTablet for tablet f6b50872bb2142fe8a7a0c2e234d24af (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:23:01.820484  1390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b50872bb2142fe8a7a0c2e234d24af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.821064  1478 raft_consensus.cc:2454] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
W20240426 13:23:01.822774  1367 consensus_peers.cc:485] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:01.823534  1537 tablet_service.cc:1467] Processing CreateTablet for tablet e266712f988c42319d640fa890ec95d5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:23:01.824149  1540 tablet_service.cc:1467] Processing CreateTablet for tablet f8cbfde04d874c1bb4aac75641436065 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:23:01.824810  1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e266712f988c42319d640fa890ec95d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.831599  1680 leader_election.cc:288] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:01.832376  1626 raft_consensus.cc:2379] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:01.832726  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:01.833202  1680 consensus_queue.cc:1035] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:01.835052  1703 consensus_queue.cc:1035] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:01.821296  1672 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:01.837486  1718 leader_election.cc:288] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:01.839939  1672 ts_tablet_manager.cc:1367] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.256s	user 0.014s	sys 0.016s
I20240426 13:23:01.857283  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cbfde04d874c1bb4aac75641436065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.837358  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:01.860757  1477 raft_consensus.cc:3037] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:01.865787  1686 raft_consensus.cc:504] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:01.867053  1551 raft_consensus.cc:2379] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
I20240426 13:23:01.866839  1389 tablet_service.cc:1467] Processing CreateTablet for tablet e266712f988c42319d640fa890ec95d5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:23:01.868252  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e266712f988c42319d640fa890ec95d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.868351  1674 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:01.867097  1464 tablet_service.cc:1467] Processing CreateTablet for tablet 80c15669d4654224aede733170c021f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:23:01.869064  1674 ts_tablet_manager.cc:1367] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.435s	user 0.008s	sys 0.009s
I20240426 13:23:01.872427  1673 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:01.877671  1386 tablet_service.cc:1467] Processing CreateTablet for tablet 80c15669d4654224aede733170c021f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:23:01.880633  1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c15669d4654224aede733170c021f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.869413  1466 tablet_service.cc:1467] Processing CreateTablet for tablet f8cbfde04d874c1bb4aac75641436065 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:23:01.882411  1387 tablet_service.cc:1467] Processing CreateTablet for tablet 4b081604971346229e0323b88039b905 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:23:01.886706  1609 tablet_service.cc:1467] Processing CreateTablet for tablet 80c15669d4654224aede733170c021f7 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:23:01.892828  1613 tablet_service.cc:1467] Processing CreateTablet for tablet e266712f988c42319d640fa890ec95d5 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:23:01.892856  1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cbfde04d874c1bb4aac75641436065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.893965  1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c15669d4654224aede733170c021f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.894181  1612 tablet_service.cc:1467] Processing CreateTablet for tablet f8cbfde04d874c1bb4aac75641436065 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:23:01.899063  1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cbfde04d874c1bb4aac75641436065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.900385  1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b50872bb2142fe8a7a0c2e234d24af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.902307  1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b081604971346229e0323b88039b905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.903029  1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e266712f988c42319d640fa890ec95d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.903733  1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c15669d4654224aede733170c021f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.904007  1536 tablet_service.cc:1467] Processing CreateTablet for tablet 4b081604971346229e0323b88039b905 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:23:01.905633  1536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b081604971346229e0323b88039b905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.910183  1673 tablet_bootstrap.cc:654] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.943284  1671 tablet_bootstrap.cc:654] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:01.947901  1672 raft_consensus.cc:348] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:01.949054  1672 raft_consensus.cc:374] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.949440  1672 raft_consensus.cc:729] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.985464  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:01.986860  1540 tablet_service.cc:1467] Processing CreateTablet for tablet c7af4930353e4432961278f8d6fc493e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:23:01.988735  1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7af4930353e4432961278f8d6fc493e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:01.971248  1674 raft_consensus.cc:348] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.991909  1674 raft_consensus.cc:374] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:01.992529  1674 raft_consensus.cc:729] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:01.993604  1674 consensus_queue.cc:260] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:01.995865  1460 tablet_service.cc:1467] Processing CreateTablet for tablet 4b081604971346229e0323b88039b905 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:23:01.997720  1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b081604971346229e0323b88039b905. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.000169  1686 leader_election.cc:288] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
W20240426 13:23:02.000945  1720 log.cc:924] Time spent T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20240426 13:23:02.001096  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:02.001739  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.002377  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd95475394f4c5bb64e94326a4cf9f5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:02.003341  1626 raft_consensus.cc:2379] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:02.002466  1475 raft_consensus.cc:2454] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:01.950358  1672 consensus_queue.cc:260] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.030252  1672 ts_tablet_manager.cc:1398] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.190s	user 0.009s	sys 0.000s
I20240426 13:23:02.031641  1672 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
W20240426 13:23:02.032764  1589 leader_election.cc:341] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.044755  1517 leader_election.cc:302] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:02.046526  1686 raft_consensus.cc:2726] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.051375  1679 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.052168  1679 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.072774  1607 tablet_service.cc:1467] Processing CreateTablet for tablet e8c8c0d5fc844f21b8ed5e96206020f0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:23:02.072863  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.074115  1626 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:02.074470  1607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c8c0d5fc844f21b8ed5e96206020f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.077780  1673 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.078341  1673 ts_tablet_manager.cc:1367] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.249s	user 0.045s	sys 0.047s
I20240426 13:23:02.081321  1673 raft_consensus.cc:348] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.094939  1368 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:02.096679  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:02.098186  1366 leader_election.cc:341] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.092571  1679 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.094834  1673 raft_consensus.cc:374] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.098647  1589 leader_election.cc:302] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.100869  1477 raft_consensus.cc:2454] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:02.104753  1703 raft_consensus.cc:2781] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.106297  1703 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.106647  1703 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.100896  1686 raft_consensus.cc:482] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.101063  1673 raft_consensus.cc:729] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.109006  1686 raft_consensus.cc:504] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.116525  1718 raft_consensus.cc:2781] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.117018  1718 raft_consensus.cc:482] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.117350  1718 raft_consensus.cc:3037] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.118425  1388 tablet_service.cc:1467] Processing CreateTablet for tablet c7af4930353e4432961278f8d6fc493e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:23:02.119719  1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7af4930353e4432961278f8d6fc493e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.119382  1461 tablet_service.cc:1467] Processing CreateTablet for tablet e8c8c0d5fc844f21b8ed5e96206020f0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:23:02.121181  1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c8c0d5fc844f21b8ed5e96206020f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.125176  1718 raft_consensus.cc:504] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.127324  1718 leader_election.cc:288] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.130834  1674 ts_tablet_manager.cc:1398] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.261s	user 0.011s	sys 0.000s
I20240426 13:23:02.143720  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.144757  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:02.144644  1389 tablet_service.cc:1467] Processing CreateTablet for tablet e8c8c0d5fc844f21b8ed5e96206020f0 (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:23:02.145403  1477 raft_consensus.cc:3037] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.145962  1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c8c0d5fc844f21b8ed5e96206020f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.150005  1674 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.150663  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.158475  1721 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.158331  1610 tablet_service.cc:1467] Processing CreateTablet for tablet c7af4930353e4432961278f8d6fc493e (DEFAULT_TABLE table=test-workload [id=52be88f7fbf544a29d538d01d974bd93]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:23:02.158725  1589 leader_election.cc:302] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:02.159001  1721 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.143807  1671 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:02.160378  1671 ts_tablet_manager.cc:1367] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.368s	user 0.012s	sys 0.005s
I20240426 13:23:02.161096  1721 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.136835  1686 leader_election.cc:288] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.167986  1680 raft_consensus.cc:2781] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.168555  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.109823  1673 consensus_queue.cc:260] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
W20240426 13:23:02.169481  1368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 94022.8us
I20240426 13:23:02.183354  1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7af4930353e4432961278f8d6fc493e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:23:02.204843  1674 tablet_bootstrap.cc:654] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.208801  1673 ts_tablet_manager.cc:1398] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.130s	user 0.006s	sys 0.002s
I20240426 13:23:02.209931  1673 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.214684  1477 raft_consensus.cc:2454] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:02.223023  1671 raft_consensus.cc:348] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.226318  1671 raft_consensus.cc:374] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.226694  1671 raft_consensus.cc:729] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
W20240426 13:23:02.228176  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 69541us
W20240426 13:23:02.229074  1517 leader_election.cc:341] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:02.229331  1589 leader_election.cc:341] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.227591  1671 consensus_queue.cc:260] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.225025  1703 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.258457  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.259763  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:02.262502  1515 leader_election.cc:341] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.251508  1680 raft_consensus.cc:686] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:02.262969  1515 leader_election.cc:302] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:02.263809  1680 consensus_queue.cc:237] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.264237  1515 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.265463  1721 raft_consensus.cc:2726] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.266017  1721 raft_consensus.cc:2726] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.267091  1674 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.267611  1674 ts_tablet_manager.cc:1367] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.118s	user 0.015s	sys 0.009s
I20240426 13:23:02.270555  1674 raft_consensus.cc:348] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.271258  1674 raft_consensus.cc:374] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.271564  1674 raft_consensus.cc:729] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.264261  1672 tablet_bootstrap.cc:654] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.273856  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:02.276499  1626 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.279695  1673 tablet_bootstrap.cc:654] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.275784  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
W20240426 13:23:02.293279  1366 leader_election.cc:341] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.283144  1626 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:02.284137  1703 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.236281  1589 leader_election.cc:302] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.294939  1368 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf
W20240426 13:23:02.295495  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 64025.1us
I20240426 13:23:02.296578  1680 raft_consensus.cc:2781] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.297232  1680 raft_consensus.cc:686] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:02.274670  1671 ts_tablet_manager.cc:1398] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.114s	user 0.007s	sys 0.001s
I20240426 13:23:02.298292  1680 consensus_queue.cc:237] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.299530  1671 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:02.301645  1672 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.302201  1672 ts_tablet_manager.cc:1367] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.271s	user 0.008s	sys 0.006s
I20240426 13:23:02.286856  1674 consensus_queue.cc:260] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.287923  1682 raft_consensus.cc:482] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.304961  1682 raft_consensus.cc:504] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.306700  1674 ts_tablet_manager.cc:1398] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.005s
I20240426 13:23:02.306941  1682 leader_election.cc:288] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.289945  1730 raft_consensus.cc:482] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.307807  1674 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.307610  1730 raft_consensus.cc:504] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.309679  1730 leader_election.cc:288] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.310237  1730 raft_consensus.cc:2781] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.308593  1300 catalog_manager.cc:5484] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:02.312253  1672 raft_consensus.cc:348] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.313055  1672 raft_consensus.cc:374] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.313347  1672 raft_consensus.cc:729] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.314226  1672 consensus_queue.cc:260] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.316885  1735 raft_consensus.cc:482] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.317410  1735 raft_consensus.cc:504] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.319118  1673 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.319422  1735 leader_election.cc:288] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.319636  1673 ts_tablet_manager.cc:1367] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.110s	user 0.019s	sys 0.014s
I20240426 13:23:02.320859  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.321724  1477 raft_consensus.cc:2379] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e4ab2f7d204691a8f5a355a66a76a4 in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:02.319638  1672 ts_tablet_manager.cc:1398] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20240426 13:23:02.322690  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.323731  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.323851  1402 raft_consensus.cc:2454] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:02.324143  1673 raft_consensus.cc:348] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.324448  1626 raft_consensus.cc:2454] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:02.322919  1730 raft_consensus.cc:686] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:02.325929  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.326566  1477 raft_consensus.cc:2454] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:02.326418  1730 consensus_queue.cc:237] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.327268  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.327689  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0013296c2f6d4bec82b9277e5df43e64" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.327852  1402 raft_consensus.cc:2454] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:02.329053  1589 leader_election.cc:302] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:02.329505  1368 leader_election.cc:302] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:02.330591  1735 raft_consensus.cc:2781] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.325203  1441 leader_election.cc:302] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:02.331264  1735 raft_consensus.cc:482] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.332203  1682 raft_consensus.cc:2781] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.334000  1735 raft_consensus.cc:3037] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.334404  1682 raft_consensus.cc:482] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.334717  1671 tablet_bootstrap.cc:654] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.334805  1682 raft_consensus.cc:3037] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.330680  1703 raft_consensus.cc:3037] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.335994  1674 tablet_bootstrap.cc:654] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.324929  1673 raft_consensus.cc:374] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.344570  1673 raft_consensus.cc:729] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.323870  1672 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:02.355325  1298 catalog_manager.cc:5484] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:02.346086  1673 consensus_queue.cc:260] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.372478  1703 raft_consensus.cc:2726] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.375819  1682 raft_consensus.cc:504] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.368244  1735 raft_consensus.cc:504] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.383580  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:02.384064  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:02.384694  1625 raft_consensus.cc:2379] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:02.385324  1477 raft_consensus.cc:2379] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:02.388198  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.388943  1402 raft_consensus.cc:3037] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.389230  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160c4150b51e426ab85b80343bc29efc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.391440  1589 leader_election.cc:302] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.392719  1680 raft_consensus.cc:2726] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.393640  1735 leader_election.cc:288] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.395184  1682 leader_election.cc:288] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.399236  1671 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:02.399915  1672 tablet_bootstrap.cc:654] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.398540  1673 ts_tablet_manager.cc:1398] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.078s	user 0.022s	sys 0.018s
I20240426 13:23:02.400507  1674 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.400580  1671 ts_tablet_manager.cc:1367] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.003s
I20240426 13:23:02.401201  1674 ts_tablet_manager.cc:1367] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.094s	user 0.014s	sys 0.006s
I20240426 13:23:02.401639  1673 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.404135  1671 raft_consensus.cc:348] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.404341  1674 raft_consensus.cc:348] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.405256  1671 raft_consensus.cc:374] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.405503  1674 raft_consensus.cc:374] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.405715  1671 raft_consensus.cc:729] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.405998  1674 raft_consensus.cc:729] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.406836  1671 consensus_queue.cc:260] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.406965  1674 consensus_queue.cc:260] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.407680  1402 raft_consensus.cc:2454] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:02.408668  1673 tablet_bootstrap.cc:654] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.409013  1441 leader_election.cc:302] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:02.409658  1671 ts_tablet_manager.cc:1398] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20240426 13:23:02.410185  1674 ts_tablet_manager.cc:1398] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20240426 13:23:02.410372  1682 raft_consensus.cc:2781] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.411124  1671 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:02.411188  1682 raft_consensus.cc:686] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:02.411780  1674 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.412506  1682 consensus_queue.cc:237] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.419481  1671 tablet_bootstrap.cc:654] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.426770  1674 tablet_bootstrap.cc:654] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.434296  1672 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.434800  1672 ts_tablet_manager.cc:1367] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.111s	user 0.011s	sys 0.009s
I20240426 13:23:02.434978  1552 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.442119  1672 raft_consensus.cc:348] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.442966  1672 raft_consensus.cc:374] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.443264  1672 raft_consensus.cc:729] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.443994  1672 consensus_queue.cc:260] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.447413  1673 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.448176  1673 ts_tablet_manager.cc:1367] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.047s	user 0.027s	sys 0.004s
I20240426 13:23:02.447860  1301 catalog_manager.cc:5484] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:02.451777  1686 raft_consensus.cc:482] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.452317  1686 raft_consensus.cc:504] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.452744  1672 ts_tablet_manager.cc:1398] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.006s
I20240426 13:23:02.453778  1672 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:02.454270  1686 leader_election.cc:288] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.456470  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.457957  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:02.459903  1515 leader_election.cc:341] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:02.460659  1515 leader_election.cc:341] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.461073  1515 leader_election.cc:302] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.464505  1686 raft_consensus.cc:2726] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.465749  1672 tablet_bootstrap.cc:654] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.466925  1300 catalog_manager.cc:5484] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:02.477995  1673 raft_consensus.cc:348] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.479054  1673 raft_consensus.cc:374] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.479418  1673 raft_consensus.cc:729] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.483670  1674 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.484148  1674 ts_tablet_manager.cc:1367] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.073s	user 0.013s	sys 0.015s
I20240426 13:23:02.486225  1674 raft_consensus.cc:348] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.486934  1674 raft_consensus.cc:374] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.487213  1674 raft_consensus.cc:729] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.487946  1673 consensus_queue.cc:260] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.488029  1674 consensus_queue.cc:260] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.490252  1673 ts_tablet_manager.cc:1398] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.042s	user 0.006s	sys 0.000s
I20240426 13:23:02.490449  1674 ts_tablet_manager.cc:1398] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:23:02.491159  1673 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.491456  1674 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.497481  1674 tablet_bootstrap.cc:654] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.503116  1674 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.503563  1674 ts_tablet_manager.cc:1367] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240426 13:23:02.504981  1673 tablet_bootstrap.cc:654] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.506112  1674 raft_consensus.cc:348] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.506976  1674 raft_consensus.cc:374] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.507249  1674 raft_consensus.cc:729] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.507915  1674 consensus_queue.cc:260] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.510107  1673 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.510257  1674 ts_tablet_manager.cc:1398] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.003s
I20240426 13:23:02.510592  1673 ts_tablet_manager.cc:1367] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.000s
I20240426 13:23:02.511189  1674 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.512909  1673 raft_consensus.cc:348] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.513530  1673 raft_consensus.cc:374] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.513739  1673 raft_consensus.cc:729] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.514302  1673 consensus_queue.cc:260] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.516108  1673 ts_tablet_manager.cc:1398] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20240426 13:23:02.516922  1673 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.518606  1672 tablet_bootstrap.cc:492] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.519127  1672 ts_tablet_manager.cc:1367] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.066s	user 0.009s	sys 0.004s
I20240426 13:23:02.521790  1673 tablet_bootstrap.cc:654] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.521973  1672 raft_consensus.cc:348] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.522790  1672 raft_consensus.cc:374] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.523070  1672 raft_consensus.cc:729] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.523644  1671 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:02.524197  1671 ts_tablet_manager.cc:1367] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.113s	user 0.016s	sys 0.031s
I20240426 13:23:02.523869  1672 consensus_queue.cc:260] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.526249  1672 ts_tablet_manager.cc:1398] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20240426 13:23:02.526929  1674 tablet_bootstrap.cc:654] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.527009  1671 raft_consensus.cc:348] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.527869  1671 raft_consensus.cc:374] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.527909  1686 raft_consensus.cc:482] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.528354  1671 raft_consensus.cc:729] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.528757  1686 raft_consensus.cc:504] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.529228  1671 consensus_queue.cc:260] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.530970  1672 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:02.532513  1686 leader_election.cc:288] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.532953  1673 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.533397  1673 ts_tablet_manager.cc:1367] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.003s
I20240426 13:23:02.533682  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.533699  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.534373  1402 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.534838  1477 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.535672  1673 raft_consensus.cc:348] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.536051  1515 leader_election.cc:302] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:02.536417  1673 raft_consensus.cc:374] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.536703  1673 raft_consensus.cc:729] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.537194  1671 ts_tablet_manager.cc:1398] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20240426 13:23:02.537583  1686 raft_consensus.cc:2781] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.538023  1686 raft_consensus.cc:482] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.538246  1671 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:02.538446  1686 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.539287  1672 tablet_bootstrap.cc:654] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.542760  1673 consensus_queue.cc:260] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
W20240426 13:23:02.543416  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:02.545282  1673 ts_tablet_manager.cc:1398] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20240426 13:23:02.546211  1673 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.550388  1671 tablet_bootstrap.cc:654] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.553001  1673 tablet_bootstrap.cc:654] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.552867  1686 raft_consensus.cc:504] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.555145  1686 leader_election.cc:288] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:02.555516  1674 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.556205  1674 ts_tablet_manager.cc:1367] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.002s
I20240426 13:23:02.558140  1673 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.558514  1673 ts_tablet_manager.cc:1367] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240426 13:23:02.560232  1673 raft_consensus.cc:348] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.560981  1673 raft_consensus.cc:374] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.561234  1673 raft_consensus.cc:729] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.561882  1673 consensus_queue.cc:260] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.562455  1672 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.562842  1671 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:02.563515  1671 ts_tablet_manager.cc:1367] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.000s
I20240426 13:23:02.563562  1672 ts_tablet_manager.cc:1367] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.001s
I20240426 13:23:02.609442  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.615072  1402 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.563993  1673 ts_tablet_manager.cc:1398] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.001s
I20240426 13:23:02.616724  1735 raft_consensus.cc:482] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.618597  1673 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.618032  1735 raft_consensus.cc:504] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.609259  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:02.559623  1674 raft_consensus.cc:348] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.621335  1686 raft_consensus.cc:482] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.622216  1477 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.632665  1686 raft_consensus.cc:504] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.634861  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:02.635922  1552 raft_consensus.cc:2454] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:02.636204  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:02.637223  1403 raft_consensus.cc:2454] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:02.637900  1590 leader_election.cc:302] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:02.639302  1680 raft_consensus.cc:2781] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.639794  1680 raft_consensus.cc:482] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.640241  1680 raft_consensus.cc:3037] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.641959  1686 leader_election.cc:288] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:02.642756  1674 raft_consensus.cc:374] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.635093  1735 leader_election.cc:288] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.643061  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f376ec882d7545d8bb64b314b89dc70c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.644474  1475 raft_consensus.cc:2454] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.646549  1515 leader_election.cc:302] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:02.631731  1673 tablet_bootstrap.cc:654] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.642957  1733 raft_consensus.cc:482] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.649538  1733 raft_consensus.cc:504] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.650929  1686 raft_consensus.cc:2781] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.651530  1686 raft_consensus.cc:482] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.651861  1686 raft_consensus.cc:3037] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.660679  1672 raft_consensus.cc:348] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.661635  1672 raft_consensus.cc:374] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.662026  1672 raft_consensus.cc:729] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.662962  1672 consensus_queue.cc:260] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.663429  1733 leader_election.cc:288] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:02.663908  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:02.665454  1589 leader_election.cc:341] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.665990  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:02.667527  1590 leader_election.cc:341] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.667359  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f376ec882d7545d8bb64b314b89dc70c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.668099  1590 leader_election.cc:302] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:02.668454  1625 raft_consensus.cc:2454] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.669502  1733 raft_consensus.cc:2726] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.669661  1672 ts_tablet_manager.cc:1398] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.001s
I20240426 13:23:02.671308  1672 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:02.673838  1477 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:02.674537  1402 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:02.675341  1515 leader_election.cc:302] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:02.676641  1721 raft_consensus.cc:2781] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.677542  1672 tablet_bootstrap.cc:654] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.643492  1674 raft_consensus.cc:729] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.681896  1674 consensus_queue.cc:260] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.630700  1671 raft_consensus.cc:348] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.688999  1671 raft_consensus.cc:374] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.689373  1671 raft_consensus.cc:729] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.689850  1672 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.690398  1672 ts_tablet_manager.cc:1367] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.004s
I20240426 13:23:02.691871  1671 consensus_queue.cc:260] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.693210  1672 raft_consensus.cc:348] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.694461  1672 raft_consensus.cc:374] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.694751  1672 raft_consensus.cc:729] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.694839  1673 tablet_bootstrap.cc:492] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.695428  1673 ts_tablet_manager.cc:1367] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.077s	user 0.010s	sys 0.010s
I20240426 13:23:02.695478  1672 consensus_queue.cc:260] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.696349  1674 ts_tablet_manager.cc:1398] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.140s	user 0.007s	sys 0.003s
I20240426 13:23:02.697934  1674 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:02.693723  1721 raft_consensus.cc:686] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:02.703477  1671 ts_tablet_manager.cc:1398] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.139s	user 0.006s	sys 0.002s
I20240426 13:23:02.703414  1721 consensus_queue.cc:237] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.704985  1671 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:02.700282  1673 raft_consensus.cc:348] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.707022  1673 raft_consensus.cc:374] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.707362  1673 raft_consensus.cc:729] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.708248  1673 consensus_queue.cc:260] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.715167  1672 ts_tablet_manager.cc:1398] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.024s	user 0.010s	sys 0.003s
I20240426 13:23:02.716769  1672 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:02.719133  1674 tablet_bootstrap.cc:654] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.720623  1673 ts_tablet_manager.cc:1398] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20240426 13:23:02.721819  1673 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.723387  1733 consensus_queue.cc:1035] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:02.724195  1672 tablet_bootstrap.cc:654] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.725119  1680 raft_consensus.cc:504] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.727856  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:02.728185  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc860ae90a4860bc539207a9259897" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.728449  1552 raft_consensus.cc:3037] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.728765  1402 raft_consensus.cc:3037] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.736898  1680 leader_election.cc:288] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.740434  1298 catalog_manager.cc:5484] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:02.736598  1402 raft_consensus.cc:2454] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:02.747536  1589 leader_election.cc:302] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:02.748543  1733 consensus_queue.cc:1035] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:02.749929  1733 raft_consensus.cc:2781] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:02.750398  1733 raft_consensus.cc:686] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:02.750686  1552 raft_consensus.cc:2454] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:02.751492  1733 consensus_queue.cc:237] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.778445  1671 tablet_bootstrap.cc:654] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.781255  1766 raft_consensus.cc:482] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.782022  1766 raft_consensus.cc:504] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.786089  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:02.786631  1673 tablet_bootstrap.cc:654] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:02.787050  1475 raft_consensus.cc:2454] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.787647  1298 catalog_manager.cc:5484] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:02.801306  1766 leader_election.cc:288] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:02.805997  1682 raft_consensus.cc:482] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.806965  1682 raft_consensus.cc:504] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:02.808925  1475 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.812475  1515 leader_election.cc:302] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:02.812574  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.813767  1625 raft_consensus.cc:2454] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:02.815553  1733 raft_consensus.cc:482] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.816118  1733 raft_consensus.cc:504] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.818133  1766 raft_consensus.cc:2781] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.818588  1766 raft_consensus.cc:482] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.818908  1766 raft_consensus.cc:3037] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.823334  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b698eb433b2d49c8b17447ce4c5b28e5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:02.825250  1589 leader_election.cc:341] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.818279  1733 leader_election.cc:288] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.826490  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b698eb433b2d49c8b17447ce4c5b28e5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:02.827165  1552 raft_consensus.cc:2454] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:02.828369  1590 leader_election.cc:302] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:02.829352  1733 raft_consensus.cc:2781] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.829721  1733 raft_consensus.cc:482] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.830041  1733 raft_consensus.cc:3037] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.851763  1735 consensus_queue.cc:1035] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:02.854571  1673 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:02.855805  1673 ts_tablet_manager.cc:1367] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.134s	user 0.013s	sys 0.002s
I20240426 13:23:02.857903  1776 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.858639  1776 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.861934  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.863694  1682 leader_election.cc:288] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:02.864452  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:02.865006  1443 leader_election.cc:341] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.865245  1402 raft_consensus.cc:2454] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:02.866513  1441 leader_election.cc:302] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:02.867444  1682 raft_consensus.cc:2781] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:02.867803  1682 raft_consensus.cc:482] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:02.868153  1682 raft_consensus.cc:3037] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.871917  1686 raft_consensus.cc:504] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.865132  1703 consensus_queue.cc:1035] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240426 13:23:02.877424  1735 consensus_queue.cc:1035] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:02.877857  1674 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:02.878383  1674 ts_tablet_manager.cc:1367] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.181s	user 0.024s	sys 0.031s
I20240426 13:23:02.879258  1673 raft_consensus.cc:348] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.884632  1673 raft_consensus.cc:374] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.884718  1672 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:02.884966  1673 raft_consensus.cc:729] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.885283  1672 ts_tablet_manager.cc:1367] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.169s	user 0.045s	sys 0.046s
I20240426 13:23:02.885717  1673 consensus_queue.cc:260] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.886240  1686 leader_election.cc:288] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:02.900564  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:02.901998  1441 leader_election.cc:341] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:02.902096  1674 raft_consensus.cc:348] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.902596  1441 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:02.903188  1674 raft_consensus.cc:374] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.903645  1674 raft_consensus.cc:729] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.913996  1673 ts_tablet_manager.cc:1398] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.058s	user 0.006s	sys 0.001s
I20240426 13:23:02.916733  1673 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:02.895866  1776 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
W20240426 13:23:02.908721  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 56268.7us
I20240426 13:23:02.941597  1776 consensus_queue.cc:1035] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:02.943466  1776 raft_consensus.cc:2726] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:02.943650  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f376ec882d7545d8bb64b314b89dc70c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:02.944310  1743 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:02.944537  1625 raft_consensus.cc:3037] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.944881  1743 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:02.937817  1733 raft_consensus.cc:504] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.940575  1672 raft_consensus.cc:348] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.956511  1672 raft_consensus.cc:374] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:02.957037  1672 raft_consensus.cc:729] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:02.958240  1672 consensus_queue.cc:260] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:02.956993  1674 consensus_queue.cc:260] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:02.978009  1733 leader_election.cc:288] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.979048  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b698eb433b2d49c8b17447ce4c5b28e5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:02.981489  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b698eb433b2d49c8b17447ce4c5b28e5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:02.982240  1551 raft_consensus.cc:3037] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:02.991400  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:02.996680  1743 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:02.998543  1672 ts_tablet_manager.cc:1398] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.113s	user 0.007s	sys 0.003s
W20240426 13:23:02.964148  1441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 62187.7us
I20240426 13:23:03.009294  1674 ts_tablet_manager.cc:1398] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.131s	user 0.009s	sys 0.003s
I20240426 13:23:03.005677  1625 raft_consensus.cc:2454] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
W20240426 13:23:03.006379  1589 leader_election.cc:341] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.022898  1672 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.023828  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f376ec882d7545d8bb64b314b89dc70c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.029290  1703 consensus_queue.cc:1035] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:03.034787  1671 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.010294  1766 raft_consensus.cc:504] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.038295  1673 tablet_bootstrap.cc:654] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.032683  1477 raft_consensus.cc:3037] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.046113  1671 ts_tablet_manager.cc:1367] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.341s	user 0.011s	sys 0.060s
I20240426 13:23:03.052362  1674 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.052603  1780 raft_consensus.cc:482] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.047123  1682 raft_consensus.cc:504] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.052099  1517 leader_election.cc:302] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.052211  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:03.055256  1686 raft_consensus.cc:482] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.072353  1721 raft_consensus.cc:2781] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.070711  1686 raft_consensus.cc:504] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.073163  1721 raft_consensus.cc:686] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:03.074651  1721 consensus_queue.cc:237] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.055830  1780 raft_consensus.cc:504] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.081984  1780 leader_election.cc:288] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.084507  1686 leader_election.cc:288] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:03.085719  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:03.086884  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:03.087134  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:03.088088  1401 raft_consensus.cc:3037] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:23:03.088428  1517 leader_election.cc:341] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.091246  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:03.091917  1625 raft_consensus.cc:3037] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.062609  1477 raft_consensus.cc:2454] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.092382  1766 leader_election.cc:288] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:03.092645  1780 consensus_queue.cc:1035] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:03.095116  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:03.102380  1441 leader_election.cc:341] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.119593  1672 tablet_bootstrap.cc:654] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.119926  1671 raft_consensus.cc:348] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.120883  1671 raft_consensus.cc:374] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.121232  1671 raft_consensus.cc:729] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.098081  1674 tablet_bootstrap.cc:654] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.137396  1551 raft_consensus.cc:2454] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:03.138116  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.138815  1682 leader_election.cc:288] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.139039  1550 raft_consensus.cc:2454] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:03.140347  1590 leader_election.cc:302] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:03.140693  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c76184e5b74fe4b94b24a896f5b89d" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:03.140883  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074e277f90624d9e8754dfd0a7bb703d" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.141167  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.124933  1671 consensus_queue.cc:260] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.145920  1403 raft_consensus.cc:2454] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:03.146744  1441 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:03.147724  1515 leader_election.cc:302] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:03.148659  1721 raft_consensus.cc:2781] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.149199  1721 raft_consensus.cc:482] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.149644  1721 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.146075  1789 raft_consensus.cc:2781] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.151785  1789 raft_consensus.cc:686] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:03.152712  1789 consensus_queue.cc:237] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.155315  1298 catalog_manager.cc:5484] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.158509  1401 raft_consensus.cc:2454] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
W20240426 13:23:03.158130  1442 leader_election.cc:341] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.159150  1442 leader_election.cc:302] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:03.159421  1475 raft_consensus.cc:3037] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.161123  1780 raft_consensus.cc:2726] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:03.167409  1625 raft_consensus.cc:2454] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.145499  1551 raft_consensus.cc:3037] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.169003  1441 leader_election.cc:302] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:03.170042  1780 raft_consensus.cc:2781] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.170513  1780 raft_consensus.cc:686] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:03.171435  1780 consensus_queue.cc:237] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.168591  1517 leader_election.cc:302] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.174306  1766 raft_consensus.cc:2781] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.174880  1766 raft_consensus.cc:686] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:03.175544  1671 ts_tablet_manager.cc:1398] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.129s	user 0.006s	sys 0.001s
I20240426 13:23:03.175750  1766 consensus_queue.cc:237] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.176759  1671 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.188019  1475 raft_consensus.cc:2454] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.191478  1682 raft_consensus.cc:2726] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:03.206180  1673 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.206725  1673 ts_tablet_manager.cc:1367] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.290s	user 0.026s	sys 0.028s
I20240426 13:23:03.209699  1673 raft_consensus.cc:348] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.210450  1673 raft_consensus.cc:374] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.211249  1673 raft_consensus.cc:729] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.218230  1671 tablet_bootstrap.cc:654] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.215194  1673 consensus_queue.cc:260] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.217793  1721 raft_consensus.cc:504] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.220116  1300 catalog_manager.cc:5484] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.213156  1551 raft_consensus.cc:2454] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:03.223626  1673 ts_tablet_manager.cc:1398] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20240426 13:23:03.226369  1721 leader_election.cc:288] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:03.230708  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:03.229768  1299 catalog_manager.cc:5484] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.231245  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:03.231364  1401 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.232256  1789 consensus_queue.cc:1035] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:23:03.235101  1517 leader_election.cc:341] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.241343  1673 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
W20240426 13:23:03.242929  1793 log.cc:924] Time spent T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20240426 13:23:03.265185  1401 raft_consensus.cc:2454] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.265906  1680 consensus_queue.cc:1035] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.267936  1515 leader_election.cc:302] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:03.269636  1673 tablet_bootstrap.cc:654] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.271821  1721 raft_consensus.cc:2781] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.272361  1721 raft_consensus.cc:686] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:03.272665  1671 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.273116  1671 ts_tablet_manager.cc:1367] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.097s	user 0.018s	sys 0.000s
I20240426 13:23:03.273180  1721 consensus_queue.cc:237] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.275794  1671 raft_consensus.cc:348] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.276517  1671 raft_consensus.cc:374] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.276770  1671 raft_consensus.cc:729] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.277370  1671 consensus_queue.cc:260] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.279541  1671 ts_tablet_manager.cc:1398] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:23:03.281519  1671 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.282620  1298 catalog_manager.cc:5484] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.284427  1674 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:03.284920  1674 ts_tablet_manager.cc:1367] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.237s	user 0.010s	sys 0.015s
I20240426 13:23:03.289567  1672 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.290074  1672 ts_tablet_manager.cc:1367] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.268s	user 0.021s	sys 0.014s
I20240426 13:23:03.294194  1674 raft_consensus.cc:348] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.294948  1674 raft_consensus.cc:374] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.295212  1674 raft_consensus.cc:729] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.297726  1674 consensus_queue.cc:260] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.302909  1671 tablet_bootstrap.cc:654] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.304075  1766 consensus_queue.cc:1035] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.307065  1674 ts_tablet_manager.cc:1398] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20240426 13:23:03.307166  1673 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.307706  1673 ts_tablet_manager.cc:1367] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.067s	user 0.019s	sys 0.002s
I20240426 13:23:03.308178  1674 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.310514  1672 raft_consensus.cc:348] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.311252  1672 raft_consensus.cc:374] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.312266  1672 raft_consensus.cc:729] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.313023  1672 consensus_queue.cc:260] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.315454  1672 ts_tablet_manager.cc:1398] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.004s
I20240426 13:23:03.316716  1672 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.317859  1673 raft_consensus.cc:348] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.318584  1673 raft_consensus.cc:374] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.318840  1673 raft_consensus.cc:729] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.323092  1674 tablet_bootstrap.cc:654] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.323944  1672 tablet_bootstrap.cc:654] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.319501  1673 consensus_queue.cc:260] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.328061  1673 ts_tablet_manager.cc:1398] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20240426 13:23:03.329231  1673 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:03.330510  1721 consensus_queue.cc:1035] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:03.333250  1301 catalog_manager.cc:5484] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.351410  1671 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.351965  1671 ts_tablet_manager.cc:1367] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.071s	user 0.017s	sys 0.005s
I20240426 13:23:03.354990  1671 raft_consensus.cc:348] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.355722  1671 raft_consensus.cc:374] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.355870  1680 raft_consensus.cc:482] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.356261  1673 tablet_bootstrap.cc:654] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.356542  1680 raft_consensus.cc:504] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.357610  1671 raft_consensus.cc:729] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.358371  1671 consensus_queue.cc:260] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.358810  1680 leader_election.cc:288] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.360785  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.361286  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:03.362043  1478 raft_consensus.cc:2454] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
W20240426 13:23:03.362246  1590 leader_election.cc:341] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.362664  1671 ts_tablet_manager.cc:1398] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20240426 13:23:03.363422  1589 leader_election.cc:302] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:03.364790  1680 raft_consensus.cc:2781] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.365239  1680 raft_consensus.cc:482] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.365725  1680 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.373534  1680 raft_consensus.cc:504] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.376770  1671 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.381562  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.382495  1478 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.361613  1672 tablet_bootstrap.cc:492] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.383790  1672 ts_tablet_manager.cc:1367] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.003s
I20240426 13:23:03.386440  1672 raft_consensus.cc:348] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.387120  1672 raft_consensus.cc:374] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.387367  1672 raft_consensus.cc:729] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.388054  1672 consensus_queue.cc:260] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.390655  1674 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:03.391276  1674 ts_tablet_manager.cc:1367] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.083s	user 0.009s	sys 0.022s
I20240426 13:23:03.394239  1674 raft_consensus.cc:348] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.394976  1674 raft_consensus.cc:374] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.396119  1674 raft_consensus.cc:729] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.396811  1478 raft_consensus.cc:2454] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:03.398080  1680 leader_election.cc:288] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.398124  1589 leader_election.cc:302] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:03.399085  1680 raft_consensus.cc:2781] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.399497  1680 raft_consensus.cc:686] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:03.399521  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:03.400528  1552 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.400424  1680 consensus_queue.cc:237] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.411301  1672 ts_tablet_manager.cc:1398] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.027s	user 0.016s	sys 0.004s
I20240426 13:23:03.413128  1672 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.397578  1674 consensus_queue.cc:260] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.415443  1552 raft_consensus.cc:2454] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:03.417225  1301 catalog_manager.cc:5484] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:03.418483  1672 tablet_bootstrap.cc:654] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.420372  1674 ts_tablet_manager.cc:1398] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.029s	user 0.002s	sys 0.005s
I20240426 13:23:03.421537  1674 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.424196  1672 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.424739  1672 ts_tablet_manager.cc:1367] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240426 13:23:03.425804  1671 tablet_bootstrap.cc:654] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.426385  1673 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.426863  1673 ts_tablet_manager.cc:1367] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.098s	user 0.012s	sys 0.010s
I20240426 13:23:03.431212  1674 tablet_bootstrap.cc:654] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.431058  1672 raft_consensus.cc:348] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.431393  1673 raft_consensus.cc:348] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.431910  1672 raft_consensus.cc:374] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.432193  1673 raft_consensus.cc:374] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.432399  1672 raft_consensus.cc:729] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.432613  1673 raft_consensus.cc:729] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.433375  1672 consensus_queue.cc:260] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.433496  1673 consensus_queue.cc:260] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.437290  1673 ts_tablet_manager.cc:1398] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20240426 13:23:03.438402  1673 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:03.440709  1672 ts_tablet_manager.cc:1398] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20240426 13:23:03.442292  1672 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.443974  1673 tablet_bootstrap.cc:654] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.450769  1671 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.451308  1671 ts_tablet_manager.cc:1367] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.075s	user 0.014s	sys 0.004s
I20240426 13:23:03.453814  1671 raft_consensus.cc:348] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.454550  1671 raft_consensus.cc:374] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.454808  1671 raft_consensus.cc:729] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.456125  1671 consensus_queue.cc:260] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.456758  1672 tablet_bootstrap.cc:654] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.458619  1674 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:03.458971  1674 ts_tablet_manager.cc:1367] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.009s
I20240426 13:23:03.459244  1673 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.459738  1673 ts_tablet_manager.cc:1367] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.008s
I20240426 13:23:03.461508  1671 ts_tablet_manager.cc:1398] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20240426 13:23:03.463198  1671 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.463129  1673 raft_consensus.cc:348] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.464257  1673 raft_consensus.cc:374] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.464584  1673 raft_consensus.cc:729] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.463264  1674 raft_consensus.cc:348] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.465348  1674 raft_consensus.cc:374] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.465665  1674 raft_consensus.cc:729] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.466068  1672 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.466529  1672 ts_tablet_manager.cc:1367] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.007s
I20240426 13:23:03.467129  1674 consensus_queue.cc:260] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.466517  1673 consensus_queue.cc:260] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.469349  1672 raft_consensus.cc:348] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.469993  1672 raft_consensus.cc:374] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.470234  1672 raft_consensus.cc:729] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.471081  1671 tablet_bootstrap.cc:654] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.470248  1703 raft_consensus.cc:482] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.471756  1703 raft_consensus.cc:504] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.473299  1674 ts_tablet_manager.cc:1398] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.003s
I20240426 13:23:03.470849  1672 consensus_queue.cc:260] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.474704  1703 leader_election.cc:288] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.475595  1674 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.479530  1672 ts_tablet_manager.cc:1398] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.001s
I20240426 13:23:03.480779  1672 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.481916  1674 tablet_bootstrap.cc:654] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.474390  1673 ts_tablet_manager.cc:1398] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20240426 13:23:03.483198  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.483878  1552 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.485559  1671 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.486020  1671 ts_tablet_manager.cc:1367] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.008s
I20240426 13:23:03.487073  1672 tablet_bootstrap.cc:654] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.487066  1367 leader_election.cc:302] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.488197  1703 raft_consensus.cc:2781] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.488543  1703 raft_consensus.cc:482] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.489041  1703 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.488565  1671 raft_consensus.cc:348] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.490473  1671 raft_consensus.cc:374] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.491036  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:03.491746  1673 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:03.491755  1478 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.490965  1671 raft_consensus.cc:729] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.494575  1671 consensus_queue.cc:260] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.497114  1671 ts_tablet_manager.cc:1398] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20240426 13:23:03.498569  1671 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.500854  1721 consensus_queue.cc:1035] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.505856  1721 raft_consensus.cc:482] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.509902  1721 raft_consensus.cc:504] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.513554  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:03.515333  1515 leader_election.cc:341] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.517513  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:03.518354  1478 raft_consensus.cc:2454] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:03.520007  1515 leader_election.cc:302] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:03.521183  1686 raft_consensus.cc:2781] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.521646  1686 raft_consensus.cc:482] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.522073  1686 raft_consensus.cc:3037] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.527832  1721 leader_election.cc:288] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:03.531075  1703 raft_consensus.cc:504] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.533726  1703 leader_election.cc:288] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.535557  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:03.536362  1552 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.541592  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.542287  1478 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.542945  1672 tablet_bootstrap.cc:492] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.543481  1672 ts_tablet_manager.cc:1367] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.063s	user 0.028s	sys 0.031s
I20240426 13:23:03.506403  1673 tablet_bootstrap.cc:654] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.560695  1686 raft_consensus.cc:504] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.564101  1401 raft_consensus.cc:3037] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.564837  1672 raft_consensus.cc:348] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.565831  1672 raft_consensus.cc:374] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.566164  1672 raft_consensus.cc:729] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.567044  1672 consensus_queue.cc:260] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.569792  1672 ts_tablet_manager.cc:1398] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.000s
W20240426 13:23:03.572702  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:03.586457  1743 raft_consensus.cc:482] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.587277  1743 raft_consensus.cc:504] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.593119  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb745ab6644f4cfb9c10676dadd72f29" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:03.594075  1624 raft_consensus.cc:2454] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:03.595686  1443 leader_election.cc:302] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:03.596959  1780 raft_consensus.cc:2781] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.597446  1780 raft_consensus.cc:482] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.597857  1780 raft_consensus.cc:3037] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.598472  1703 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.589610  1478 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:03.599107  1703 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.600144  1743 leader_election.cc:288] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:03.601238  1703 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:03.602207  1686 leader_election.cc:288] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:03.574891  1672 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.605062  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:03.605765  1624 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.606518  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:03.606993  1368 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:03.607381  1475 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.602975  1671 tablet_bootstrap.cc:654] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.603682  1680 consensus_queue.cc:1035] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:03.605455  1366 leader_election.cc:302] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:03.611390  1721 raft_consensus.cc:482] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.557003  1552 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:03.618191  1721 raft_consensus.cc:504] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.625156  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.625826  1475 raft_consensus.cc:3037] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.626103  1721 leader_election.cc:288] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:03.622438  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb745ab6644f4cfb9c10676dadd72f29" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.637116  1824 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.637898  1824 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.640278  1824 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.640931  1824 raft_consensus.cc:2781] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.641335  1824 raft_consensus.cc:686] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:03.641443  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.642226  1824 consensus_queue.cc:237] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
W20240426 13:23:03.643030  1367 leader_election.cc:341] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.637202  1777 raft_consensus.cc:2781] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.643671  1777 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.650410  1674 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:03.650947  1674 ts_tablet_manager.cc:1367] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.176s	user 0.029s	sys 0.063s
I20240426 13:23:03.652645  1550 raft_consensus.cc:2454] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:03.644582  1777 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.653980  1735 consensus_queue.cc:1035] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240426 13:23:03.676151  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:03.676615  1366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 70157.3us
I20240426 13:23:03.676923  1625 raft_consensus.cc:2454] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:03.678359  1517 leader_election.cc:302] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.678905  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:03.679577  1477 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.680398  1766 consensus_queue.cc:1035] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.681452  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb1fddf692c4843acc3c5f994fdf53e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:03.682718  1672 tablet_bootstrap.cc:654] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.683066  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:03.686033  1673 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.686682  1673 ts_tablet_manager.cc:1367] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.195s	user 0.026s	sys 0.039s
I20240426 13:23:03.681859  1766 raft_consensus.cc:2781] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.695708  1674 raft_consensus.cc:348] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.696979  1674 raft_consensus.cc:374] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.697194  1674 raft_consensus.cc:729] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.699091  1674 consensus_queue.cc:260] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.701073  1366 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:03.701994  1828 raft_consensus.cc:2781] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.702386  1828 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.702909  1828 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.696462  1766 raft_consensus.cc:482] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.703447  1766 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:23:03.703555  1515 leader_election.cc:341] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:03.704378  1515 leader_election.cc:341] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.706806  1674 ts_tablet_manager.cc:1398] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.001s
I20240426 13:23:03.707999  1674 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.708825  1826 consensus_queue.cc:1035] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:03.710963  1828 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.711917  1743 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.712556  1475 raft_consensus.cc:2454] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.714677  1766 raft_consensus.cc:504] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.712554  1743 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.718334  1743 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:03.718220  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:03.714217  1780 raft_consensus.cc:504] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.729101  1671 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.729187  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:03.729609  1515 leader_election.cc:302] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:03.730080  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:03.721982  1625 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:23:03.730840  1443 leader_election.cc:341] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:03.731403  1441 leader_election.cc:341] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.731684  1671 ts_tablet_manager.cc:1367] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.233s	user 0.014s	sys 0.007s
I20240426 13:23:03.731853  1441 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:03.733160  1826 raft_consensus.cc:2781] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.733707  1826 raft_consensus.cc:686] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:03.735054  1743 raft_consensus.cc:2726] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:03.734828  1671 raft_consensus.cc:348] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.735798  1671 raft_consensus.cc:374] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.734546  1826 consensus_queue.cc:237] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.736155  1671 raft_consensus.cc:729] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.737241  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb745ab6644f4cfb9c10676dadd72f29" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:03.738265  1673 raft_consensus.cc:348] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.741604  1552 raft_consensus.cc:3037] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.737102  1827 consensus_queue.cc:1035] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240426 13:23:03.746678  1828 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:03.746505  1300 catalog_manager.cc:5484] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.747648  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:03.747964  1682 consensus_queue.cc:1035] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.748766  1674 tablet_bootstrap.cc:654] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
W20240426 13:23:03.749173  1367 leader_election.cc:341] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.737896  1671 consensus_queue.cc:260] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.751681  1673 raft_consensus.cc:374] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.753376  1673 raft_consensus.cc:729] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.752282  1625 raft_consensus.cc:2454] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.717689  1777 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.759971  1673 consensus_queue.cc:260] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.758278  1766 leader_election.cc:288] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:03.756820  1780 leader_election.cc:288] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:03.766300  1517 leader_election.cc:302] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.766870  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:03.767127  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb745ab6644f4cfb9c10676dadd72f29" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:03.767493  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.767521  1402 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.768055  1625 raft_consensus.cc:3037] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.768122  1477 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.772399  1671 ts_tablet_manager.cc:1398] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.003s
I20240426 13:23:03.773398  1671 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.776635  1552 raft_consensus.cc:2454] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:03.778868  1671 tablet_bootstrap.cc:654] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.781639  1766 raft_consensus.cc:2781] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.782968  1766 raft_consensus.cc:686] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:03.783941  1766 consensus_queue.cc:237] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.786582  1827 consensus_queue.cc:1035] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:23:03.802834  1477 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:03.801398  1673 ts_tablet_manager.cc:1398] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.114s	user 0.008s	sys 0.000s
I20240426 13:23:03.806895  1673 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:03.808056  1672 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:03.808961  1672 ts_tablet_manager.cc:1367] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.234s	user 0.016s	sys 0.006s
I20240426 13:23:03.809420  1777 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:03.811157  1402 raft_consensus.cc:2454] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:03.811605  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:03.812531  1626 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.815644  1442 leader_election.cc:302] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.816834  1682 raft_consensus.cc:2781] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.817387  1682 raft_consensus.cc:686] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:03.814103  1672 raft_consensus.cc:348] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.818287  1682 consensus_queue.cc:237] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.825253  1623 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.842398  1298 catalog_manager.cc:5484] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:03.828958  1780 consensus_queue.cc:1035] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.832808  1625 raft_consensus.cc:2454] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:03.843879  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:03.844755  1477 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.844214  1298 catalog_manager.cc:5484] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.846822  1672 raft_consensus.cc:374] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.847154  1672 raft_consensus.cc:729] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.834957  1673 tablet_bootstrap.cc:654] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.854395  1826 consensus_queue.cc:1035] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.837610  1671 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.838070  1626 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:03.865397  1671 ts_tablet_manager.cc:1367] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.092s	user 0.015s	sys 0.010s
I20240426 13:23:03.865866  1368 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:03.868433  1777 raft_consensus.cc:2781] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.869004  1777 raft_consensus.cc:686] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:03.869832  1777 consensus_queue.cc:237] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.872803  1671 raft_consensus.cc:348] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.876366  1671 raft_consensus.cc:374] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.877164  1671 raft_consensus.cc:729] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.878352  1671 consensus_queue.cc:260] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.882210  1671 ts_tablet_manager.cc:1398] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.001s
I20240426 13:23:03.885056  1671 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.886812  1673 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.888904  1673 ts_tablet_manager.cc:1367] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.082s	user 0.013s	sys 0.005s
I20240426 13:23:03.896659  1366 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:03.899511  1828 raft_consensus.cc:2781] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:03.900060  1828 raft_consensus.cc:686] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:03.900599  1674 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:03.901149  1674 ts_tablet_manager.cc:1367] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.193s	user 0.014s	sys 0.016s
I20240426 13:23:03.900902  1828 consensus_queue.cc:237] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:03.899045  1672 consensus_queue.cc:260] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.904157  1477 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:03.904402  1671 tablet_bootstrap.cc:654] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.904175  1674 raft_consensus.cc:348] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.908371  1674 raft_consensus.cc:374] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.908679  1674 raft_consensus.cc:729] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.909356  1674 consensus_queue.cc:260] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.912297  1672 ts_tablet_manager.cc:1398] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.103s	user 0.007s	sys 0.001s
I20240426 13:23:03.913869  1672 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:03.919348  1673 raft_consensus.cc:348] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.920244  1673 raft_consensus.cc:374] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.920528  1673 raft_consensus.cc:729] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.928627  1300 catalog_manager.cc:5484] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.935032  1671 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.935622  1671 ts_tablet_manager.cc:1367] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.051s	user 0.007s	sys 0.010s
I20240426 13:23:03.938007  1673 consensus_queue.cc:260] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:03.939548  1671 raft_consensus.cc:348] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.940295  1673 ts_tablet_manager.cc:1398] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.000s
I20240426 13:23:03.940340  1671 raft_consensus.cc:374] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.941004  1671 raft_consensus.cc:729] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.941743  1671 consensus_queue.cc:260] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.941932  1301 catalog_manager.cc:5484] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:03.944293  1671 ts_tablet_manager.cc:1398] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:23:03.944859  1673 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:03.945346  1671 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:03.947671  1674 ts_tablet_manager.cc:1398] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.046s	user 0.005s	sys 0.000s
I20240426 13:23:03.948773  1674 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:03.950147  1672 tablet_bootstrap.cc:654] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.957118  1671 tablet_bootstrap.cc:654] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.962054  1735 consensus_queue.cc:1035] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.963788  1674 tablet_bootstrap.cc:654] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.966167  1735 consensus_queue.cc:1035] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:03.970047  1777 raft_consensus.cc:482] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:03.970721  1777 raft_consensus.cc:504] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:03.972858  1777 leader_election.cc:288] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:03.975133  1673 tablet_bootstrap.cc:654] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:03.975338  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:03.976001  1552 raft_consensus.cc:2454] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:03.978096  1367 leader_election.cc:302] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:03.986696  1671 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:03.986771  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:03.987372  1671 ts_tablet_manager.cc:1367] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.042s	user 0.016s	sys 0.002s
I20240426 13:23:03.988024  1777 raft_consensus.cc:2781] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:03.988523  1777 raft_consensus.cc:482] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:03.991626  1777 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:03.993654  1673 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:03.994184  1673 ts_tablet_manager.cc:1367] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.050s	user 0.010s	sys 0.000s
I20240426 13:23:03.990024  1671 raft_consensus.cc:348] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.990502  1301 catalog_manager.cc:5484] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
W20240426 13:23:03.989390  1368 leader_election.cc:341] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:03.996018  1671 raft_consensus.cc:374] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.996392  1671 raft_consensus.cc:729] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.996923  1673 raft_consensus.cc:348] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.997026  1671 consensus_queue.cc:260] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.997664  1673 raft_consensus.cc:374] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:03.997977  1673 raft_consensus.cc:729] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:03.998633  1673 consensus_queue.cc:260] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:03.999215  1671 ts_tablet_manager.cc:1398] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.002s
I20240426 13:23:04.000866  1671 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.000988  1672 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.001492  1672 ts_tablet_manager.cc:1367] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.088s	user 0.009s	sys 0.008s
I20240426 13:23:04.003934  1672 raft_consensus.cc:348] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.004633  1672 raft_consensus.cc:374] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.004945  1672 raft_consensus.cc:729] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.006108  1672 consensus_queue.cc:260] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.007999  1672 ts_tablet_manager.cc:1398] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240426 13:23:04.010955  1672 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.011689  1673 ts_tablet_manager.cc:1398] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.002s
I20240426 13:23:04.012403  1674 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.012943  1674 ts_tablet_manager.cc:1367] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.064s	user 0.007s	sys 0.007s
I20240426 13:23:04.012890  1673 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.017751  1672 tablet_bootstrap.cc:654] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.018030  1671 tablet_bootstrap.cc:654] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.023293  1674 raft_consensus.cc:348] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.024616  1674 raft_consensus.cc:374] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.024938  1674 raft_consensus.cc:729] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.025784  1674 consensus_queue.cc:260] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.028247  1674 ts_tablet_manager.cc:1398] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20240426 13:23:04.029616  1674 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.035480  1828 raft_consensus.cc:482] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.036216  1828 raft_consensus.cc:504] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.037128  1673 tablet_bootstrap.cc:654] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.040326  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.041271  1478 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:04.042975  1777 raft_consensus.cc:504] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.044734  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.045392  1624 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:04.046685  1828 leader_election.cc:288] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.046838  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:04.043056  1366 leader_election.cc:302] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:04.047701  1551 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.050297  1828 raft_consensus.cc:2781] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.050784  1828 raft_consensus.cc:482] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.051172  1828 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.052313  1672 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.052829  1672 ts_tablet_manager.cc:1367] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.042s	user 0.010s	sys 0.006s
I20240426 13:23:04.055410  1672 raft_consensus.cc:348] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.056104  1672 raft_consensus.cc:374] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.056278  1674 tablet_bootstrap.cc:654] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.056382  1672 raft_consensus.cc:729] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.059372  1672 consensus_queue.cc:260] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.060180  1671 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.060655  1671 ts_tablet_manager.cc:1367] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.060s	user 0.006s	sys 0.037s
I20240426 13:23:04.061295  1672 ts_tablet_manager.cc:1398] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:23:04.062351  1828 raft_consensus.cc:504] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.063364  1671 raft_consensus.cc:348] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.064167  1671 raft_consensus.cc:374] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.064448  1671 raft_consensus.cc:729] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.064479  1551 raft_consensus.cc:2454] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:04.065263  1671 consensus_queue.cc:260] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.065735  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:04.066377  1624 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.062482  1672 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.073048  1672 tablet_bootstrap.cc:654] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.073825  1624 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:04.075060  1368 leader_election.cc:302] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:04.075940  1824 raft_consensus.cc:2781] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.076458  1824 raft_consensus.cc:686] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:04.077292  1824 consensus_queue.cc:237] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.080715  1671 ts_tablet_manager.cc:1398] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.004s
I20240426 13:23:04.080899  1367 leader_election.cc:302] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.081773  1825 raft_consensus.cc:2781] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.082054  1671 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.082314  1825 raft_consensus.cc:686] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:04.083155  1825 consensus_queue.cc:237] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.088371  1671 tablet_bootstrap.cc:654] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.088516  1682 raft_consensus.cc:482] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.089015  1682 raft_consensus.cc:504] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.091660  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:04.093021  1442 leader_election.cc:341] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.096720  1777 leader_election.cc:288] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.093621  1721 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.098712  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:04.099360  1721 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.101506  1721 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
W20240426 13:23:04.101794  1368 leader_election.cc:341] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.103497  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:04.105895  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.106889  1478 raft_consensus.cc:2379] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in this term.
I20240426 13:23:04.108386  1515 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:04.110459  1828 leader_election.cc:288] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.110554  1766 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.112318  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:04.112977  1478 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.113724  1671 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.114094  1671 ts_tablet_manager.cc:1367] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.032s	user 0.008s	sys 0.014s
I20240426 13:23:04.110558  1673 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.116653  1673 ts_tablet_manager.cc:1367] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.104s	user 0.014s	sys 0.000s
I20240426 13:23:04.116540  1671 raft_consensus.cc:348] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.116812  1682 leader_election.cc:288] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.117475  1671 raft_consensus.cc:374] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.117767  1671 raft_consensus.cc:729] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.118249  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.118412  1671 consensus_queue.cc:260] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.119421  1673 raft_consensus.cc:348] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
W20240426 13:23:04.119815  1443 leader_election.cc:341] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.120151  1673 raft_consensus.cc:374] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.120486  1673 raft_consensus.cc:729] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.120335  1443 leader_election.cc:302] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:04.121031  1671 ts_tablet_manager.cc:1398] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20240426 13:23:04.121421  1478 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:04.121784  1682 raft_consensus.cc:2726] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.122186  1671 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.125665  1766 raft_consensus.cc:2726] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.121415  1673 consensus_queue.cc:260] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.128173  1673 ts_tablet_manager.cc:1398] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20240426 13:23:04.130673  1671 tablet_bootstrap.cc:654] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.134810  1673 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.141979  1672 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.142540  1672 ts_tablet_manager.cc:1367] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.080s	user 0.011s	sys 0.009s
I20240426 13:23:04.144253  1300 catalog_manager.cc:5484] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:04.145197  1672 raft_consensus.cc:348] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.145747  1672 raft_consensus.cc:374] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.145949  1672 raft_consensus.cc:729] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.146132  1671 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.146623  1671 ts_tablet_manager.cc:1367] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.004s
I20240426 13:23:04.146504  1672 consensus_queue.cc:260] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.147607  1674 tablet_bootstrap.cc:492] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.148092  1674 ts_tablet_manager.cc:1367] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.119s	user 0.011s	sys 0.007s
I20240426 13:23:04.149356  1671 raft_consensus.cc:348] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.150054  1671 raft_consensus.cc:374] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.150298  1671 raft_consensus.cc:729] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.150506  1674 raft_consensus.cc:348] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.151237  1674 raft_consensus.cc:374] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.151058  1671 consensus_queue.cc:260] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.151495  1674 raft_consensus.cc:729] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.152276  1674 consensus_queue.cc:260] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.148886  1672 ts_tablet_manager.cc:1398] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20240426 13:23:04.153537  1671 ts_tablet_manager.cc:1398] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240426 13:23:04.154583  1671 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.154619  1672 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.156862  1673 tablet_bootstrap.cc:654] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.159286  1721 consensus_queue.cc:1035] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.160362  1671 tablet_bootstrap.cc:654] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.164788  1824 consensus_queue.cc:1035] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:04.166263  1721 raft_consensus.cc:482] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.166769  1721 raft_consensus.cc:504] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.168792  1721 leader_election.cc:288] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.170301  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.171020  1477 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:04.172379  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.173048  1624 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:04.174463  1515 leader_election.cc:302] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.175551  1826 raft_consensus.cc:2781] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.175904  1826 raft_consensus.cc:482] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.176266  1826 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.177861  1298 catalog_manager.cc:5484] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:04.182549  1674 ts_tablet_manager.cc:1398] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.000s
I20240426 13:23:04.193904  1682 raft_consensus.cc:482] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.194511  1682 raft_consensus.cc:504] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.197613  1674 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.197795  1682 leader_election.cc:288] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:04.198802  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:04.200337  1441 leader_election.cc:341] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.200678  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.206862  1674 tablet_bootstrap.cc:654] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.205211  1826 raft_consensus.cc:504] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
W20240426 13:23:04.209995  1443 leader_election.cc:341] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.210494  1443 leader_election.cc:302] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:04.211290  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:04.211962  1475 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.216449  1682 raft_consensus.cc:2726] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.220860  1475 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:04.221613  1674 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.222312  1674 ts_tablet_manager.cc:1367] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.009s
I20240426 13:23:04.222277  1515 leader_election.cc:302] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.223466  1721 raft_consensus.cc:2781] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.224203  1721 raft_consensus.cc:686] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:04.225762  1721 consensus_queue.cc:237] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.227177  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:04.227941  1624 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.237336  1674 raft_consensus.cc:348] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.238445  1674 raft_consensus.cc:374] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.238673  1674 raft_consensus.cc:729] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.239784  1674 consensus_queue.cc:260] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.242205  1674 ts_tablet_manager.cc:1398] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20240426 13:23:04.243592  1674 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.244302  1672 tablet_bootstrap.cc:654] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.232602  1826 leader_election.cc:288] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.256111  1400 raft_consensus.cc:3037] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.259476  1300 catalog_manager.cc:5484] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:04.275976  1674 tablet_bootstrap.cc:654] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.277597  1824 consensus_queue.cc:1035] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.294364  1624 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:04.301014  1671 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.301540  1671 ts_tablet_manager.cc:1367] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.147s	user 0.056s	sys 0.043s
I20240426 13:23:04.306491  1673 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.306999  1673 ts_tablet_manager.cc:1367] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.172s	user 0.010s	sys 0.001s
I20240426 13:23:04.307847  1671 raft_consensus.cc:348] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.308636  1671 raft_consensus.cc:374] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.308876  1671 raft_consensus.cc:729] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.309509  1671 consensus_queue.cc:260] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.310776  1766 consensus_queue.cc:1035] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.311609  1671 ts_tablet_manager.cc:1398] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20240426 13:23:04.314121  1673 raft_consensus.cc:348] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.314860  1673 raft_consensus.cc:374] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.315279  1673 raft_consensus.cc:729] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.316052  1673 consensus_queue.cc:260] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.318445  1673 ts_tablet_manager.cc:1398] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.003s
I20240426 13:23:04.319427  1673 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.312609  1671 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.323240  1672 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.323737  1672 ts_tablet_manager.cc:1367] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.169s	user 0.021s	sys 0.013s
I20240426 13:23:04.326339  1672 raft_consensus.cc:348] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.327003  1672 raft_consensus.cc:374] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.327252  1824 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.332891  1671 tablet_bootstrap.cc:654] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.333114  1824 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.327275  1672 raft_consensus.cc:729] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.337967  1824 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.336759  1672 consensus_queue.cc:260] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.339107  1674 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.339579  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.340072  1672 ts_tablet_manager.cc:1398] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.003s
I20240426 13:23:04.340595  1478 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:04.342317  1824 raft_consensus.cc:482] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.342487  1366 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:04.342849  1824 raft_consensus.cc:504] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.343669  1828 raft_consensus.cc:2781] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.352713  1828 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.353081  1828 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.339597  1674 ts_tablet_manager.cc:1367] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.096s	user 0.024s	sys 0.020s
I20240426 13:23:04.360430  1828 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.346453  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.347820  1766 consensus_queue.cc:1035] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.363286  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
W20240426 13:23:04.364740  1368 leader_election.cc:341] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.365746  1673 tablet_bootstrap.cc:654] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
W20240426 13:23:04.368328  1368 leader_election.cc:341] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.369815  1682 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.370301  1682 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.370996  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.371636  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.372323  1682 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
W20240426 13:23:04.372560  1367 leader_election.cc:341] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:04.373081  1368 leader_election.cc:341] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.373193  1674 raft_consensus.cc:348] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.373554  1368 leader_election.cc:302] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:04.349749  1672 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.373922  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:04.374354  1824 leader_election.cc:288] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.374706  1703 raft_consensus.cc:2726] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.375295  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.373965  1674 raft_consensus.cc:374] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.376363  1674 raft_consensus.cc:729] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
W20240426 13:23:04.376879  1442 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.377128  1674 consensus_queue.cc:260] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.377962  1703 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.378221  1733 raft_consensus.cc:482] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.378763  1733 raft_consensus.cc:504] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.379472  1674 ts_tablet_manager.cc:1398] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.002s
I20240426 13:23:04.380447  1674 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.380664  1733 leader_election.cc:288] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:04.383471  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.389597  1552 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:04.396435  1766 raft_consensus.cc:482] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.411464  1766 raft_consensus.cc:504] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.412806  1682 consensus_queue.cc:1035] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.398730  1703 raft_consensus.cc:482] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.415212  1703 raft_consensus.cc:504] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.418185  1777 consensus_queue.cc:1035] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.419405  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:04.420100  1402 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:04.420924  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:04.391486  1441 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.421811  1549 raft_consensus.cc:2454] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:04.422286  1441 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:04.423421  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.424263  1743 raft_consensus.cc:2726] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.424225  1475 raft_consensus.cc:2454] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:04.425047  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:04.425657  1403 raft_consensus.cc:2454] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:04.427500  1672 tablet_bootstrap.cc:654] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.434469  1515 leader_election.cc:302] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.434923  1366 leader_election.cc:302] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:04.435334  1721 raft_consensus.cc:2781] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.435698  1721 raft_consensus.cc:482] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.435765  1825 raft_consensus.cc:2781] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.436144  1721 raft_consensus.cc:3037] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.436206  1825 raft_consensus.cc:482] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.436684  1825 raft_consensus.cc:3037] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.390389  1779 raft_consensus.cc:482] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.447818  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:04.448503  1475 raft_consensus.cc:2454] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:04.395923  1826 consensus_queue.cc:1035] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20240426 13:23:04.453965  1590 leader_election.cc:302] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.456311  1680 raft_consensus.cc:2781] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.456702  1680 raft_consensus.cc:482] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.457047  1680 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.459309  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:04.460125  1475 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.444058  1779 raft_consensus.cc:504] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.464074  1779 leader_election.cc:288] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:04.465360  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.465807  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:04.467075  1590 leader_election.cc:341] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:04.467801  1589 leader_election.cc:341] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.468358  1589 leader_election.cc:302] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:04.418385  1703 leader_election.cc:288] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:04.433924  1828 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.482501  1674 tablet_bootstrap.cc:654] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.473555  1779 raft_consensus.cc:2726] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.451565  1721 raft_consensus.cc:504] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.484021  1777 consensus_queue.cc:1035] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:04.485723  1743 consensus_queue.cc:1035] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.486429  1671 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.438673  1766 leader_election.cc:288] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.482270  1825 raft_consensus.cc:504] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.494743  1671 ts_tablet_manager.cc:1367] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.182s	user 0.036s	sys 0.033s
I20240426 13:23:04.495783  1721 leader_election.cc:288] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:04.507930  1777 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:04.517385  1626 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.533596  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:04.534304  1477 raft_consensus.cc:3037] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.537592  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:04.538954  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:04.539778  1400 raft_consensus.cc:2379] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in this term.
I20240426 13:23:04.545506  1680 raft_consensus.cc:504] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.547559  1680 leader_election.cc:288] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:04.548065  1477 raft_consensus.cc:2454] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:04.548542  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:04.549342  1551 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.572396  1671 raft_consensus.cc:348] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.573480  1671 raft_consensus.cc:374] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.573838  1671 raft_consensus.cc:729] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.549175  1515 leader_election.cc:302] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:04.575735  1671 consensus_queue.cc:260] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.589414  1672 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.589852  1672 ts_tablet_manager.cc:1367] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.241s	user 0.013s	sys 0.002s
I20240426 13:23:04.576269  1825 leader_election.cc:288] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:04.594738  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612eff866f5b4a9e902f703f3f11eb82" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:04.600879  1826 raft_consensus.cc:2781] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.601459  1826 raft_consensus.cc:686] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:04.602357  1826 consensus_queue.cc:237] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.603966  1671 ts_tablet_manager.cc:1398] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.109s	user 0.006s	sys 0.006s
W20240426 13:23:04.605866  1515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 56776.6us
I20240426 13:23:04.607897  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:04.608585  1400 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.614912  1828 consensus_queue.cc:1035] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.615959  1703 consensus_queue.cc:1035] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.567014  1475 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:04.622081  1366 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:04.637650  1673 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.638198  1673 ts_tablet_manager.cc:1367] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.319s	user 0.011s	sys 0.028s
I20240426 13:23:04.639953  1672 raft_consensus.cc:348] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.640753  1672 raft_consensus.cc:374] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.627930  1824 raft_consensus.cc:2781] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.641031  1672 raft_consensus.cc:729] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.645900  1671 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.643720  1672 consensus_queue.cc:260] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.649596  1674 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.650004  1674 ts_tablet_manager.cc:1367] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.270s	user 0.015s	sys 0.022s
I20240426 13:23:04.653718  1400 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:04.655016  1589 leader_election.cc:302] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:04.655937  1680 raft_consensus.cc:2781] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.656453  1680 raft_consensus.cc:686] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:04.657395  1673 raft_consensus.cc:348] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.657975  1673 raft_consensus.cc:374] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.658181  1673 raft_consensus.cc:729] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.645352  1824 raft_consensus.cc:686] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:04.660691  1673 consensus_queue.cc:260] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.669271  1367 leader_election.cc:302] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:04.670905  1828 raft_consensus.cc:2726] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 13:23:04.673725  1680 consensus_queue.cc:237] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:04.674209  1828 consensus_queue.cc:1035] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.676748  1824 consensus_queue.cc:237] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.680217  1880 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.682374  1672 ts_tablet_manager.cc:1398] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.092s	user 0.005s	sys 0.002s
I20240426 13:23:04.682058  1674 raft_consensus.cc:348] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.682821  1674 raft_consensus.cc:374] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.683166  1674 raft_consensus.cc:729] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.660091  1551 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:04.685532  1300 catalog_manager.cc:5484] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:04.689289  1671 tablet_bootstrap.cc:654] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.683355  1672 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.683820  1674 consensus_queue.cc:260] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.703125  1673 ts_tablet_manager.cc:1398] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.003s
W20240426 13:23:04.713131  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20240426 13:23:04.719147  1885 log.cc:924] Time spent T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: Append to log took a long time: real 0.088s	user 0.003s	sys 0.000s
I20240426 13:23:04.722164  1672 tablet_bootstrap.cc:654] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.725703  1673 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.728101  1301 catalog_manager.cc:5484] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:04.730126  1671 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.730522  1671 ts_tablet_manager.cc:1367] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.000s
I20240426 13:23:04.735998  1777 consensus_queue.cc:1035] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:04.745766  1674 ts_tablet_manager.cc:1398] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.096s	user 0.007s	sys 0.000s
I20240426 13:23:04.751206  1671 raft_consensus.cc:348] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.752027  1671 raft_consensus.cc:374] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.752336  1671 raft_consensus.cc:729] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.754679  1671 consensus_queue.cc:260] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.760680  1673 tablet_bootstrap.cc:654] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.762044  1674 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.771257  1880 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:04.781749  1671 ts_tablet_manager.cc:1398] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.000s
I20240426 13:23:04.793689  1674 tablet_bootstrap.cc:654] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.806128  1671 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.819384  1671 tablet_bootstrap.cc:654] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.824494  1672 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.825017  1672 ts_tablet_manager.cc:1367] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.142s	user 0.015s	sys 0.008s
I20240426 13:23:04.827577  1672 raft_consensus.cc:348] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.828310  1672 raft_consensus.cc:374] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.828590  1672 raft_consensus.cc:729] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.829277  1672 consensus_queue.cc:260] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.834349  1673 tablet_bootstrap.cc:492] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.834877  1673 ts_tablet_manager.cc:1367] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.110s	user 0.011s	sys 0.010s
I20240426 13:23:04.837819  1673 raft_consensus.cc:348] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.838580  1673 raft_consensus.cc:374] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.838915  1673 raft_consensus.cc:729] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.839085  1672 ts_tablet_manager.cc:1398] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20240426 13:23:04.842729  1673 consensus_queue.cc:260] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.844130  1672 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.847584  1297 catalog_manager.cc:5484] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:04.850755  1673 ts_tablet_manager.cc:1398] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20240426 13:23:04.851949  1673 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.860173  1682 raft_consensus.cc:482] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.860335  1672 tablet_bootstrap.cc:654] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.860709  1682 raft_consensus.cc:504] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.862732  1682 leader_election.cc:288] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.864562  1682 raft_consensus.cc:482] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:04.865039  1682 raft_consensus.cc:504] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.865352  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.866019  1549 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:04.867138  1442 leader_election.cc:302] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.867761  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289eab6dc655403aa7b8a801972d0eaf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:04.868023  1743 raft_consensus.cc:2781] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.868487  1743 raft_consensus.cc:482] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.868659  1671 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.868860  1743 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.869196  1671 ts_tablet_manager.cc:1367] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.087s	user 0.023s	sys 0.007s
I20240426 13:23:04.870687  1674 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:04.871184  1674 ts_tablet_manager.cc:1367] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.109s	user 0.013s	sys 0.012s
I20240426 13:23:04.871737  1671 raft_consensus.cc:348] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.872509  1671 raft_consensus.cc:374] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.872761  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.873667  1622 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:04.868511  1549 raft_consensus.cc:2454] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:04.873679  1674 raft_consensus.cc:348] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.876271  1674 raft_consensus.cc:374] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.877112  1442 leader_election.cc:302] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:04.876515  1674 raft_consensus.cc:729] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.877457  1672 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:04.878103  1672 ts_tablet_manager.cc:1367] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.003s
I20240426 13:23:04.878093  1890 raft_consensus.cc:2781] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:04.878939  1890 raft_consensus.cc:482] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:04.878218  1674 consensus_queue.cc:260] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.879257  1682 leader_election.cc:288] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.879297  1890 raft_consensus.cc:3037] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.880563  1743 raft_consensus.cc:504] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.881263  1672 raft_consensus.cc:348] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.881922  1672 raft_consensus.cc:374] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.882236  1672 raft_consensus.cc:729] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.882499  1743 leader_election.cc:288] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.883296  1673 tablet_bootstrap.cc:654] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.882849  1672 consensus_queue.cc:260] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.872802  1671 raft_consensus.cc:729] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.886500  1671 consensus_queue.cc:260] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.890477  1674 ts_tablet_manager.cc:1398] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20240426 13:23:04.891474  1674 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:04.893188  1672 ts_tablet_manager.cc:1398] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20240426 13:23:04.894505  1672 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:04.895321  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:04.895974  1626 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.897965  1674 tablet_bootstrap.cc:654] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.900388  1671 ts_tablet_manager.cc:1398] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20240426 13:23:04.901393  1671 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.903712  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:04.904369  1549 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.908819  1671 tablet_bootstrap.cc:654] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.909121  1673 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.909580  1673 ts_tablet_manager.cc:1367] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.058s	user 0.022s	sys 0.004s
I20240426 13:23:04.912181  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289eab6dc655403aa7b8a801972d0eaf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:04.911545  1890 raft_consensus.cc:504] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.914716  1890 leader_election.cc:288] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:04.915927  1673 raft_consensus.cc:348] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.917832  1673 raft_consensus.cc:374] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.918187  1673 raft_consensus.cc:729] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.920179  1673 consensus_queue.cc:260] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.921361  1626 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:04.922261  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289eab6dc655403aa7b8a801972d0eaf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:04.922923  1551 raft_consensus.cc:3037] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:04.923452  1673 ts_tablet_manager.cc:1398] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20240426 13:23:04.924470  1673 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.925391  1672 tablet_bootstrap.cc:654] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.930752  1673 tablet_bootstrap.cc:654] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.940740  1671 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:04.941296  1671 ts_tablet_manager.cc:1367] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.040s	user 0.009s	sys 0.008s
I20240426 13:23:04.953228  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289eab6dc655403aa7b8a801972d0eaf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:04.957098  1549 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
W20240426 13:23:04.950678  1443 leader_election.cc:341] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.961335  1443 leader_election.cc:302] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:04.962419  1890 raft_consensus.cc:2781] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:23:04.962625  1443 leader_election.cc:341] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:04.962917  1890 raft_consensus.cc:686] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:04.963790  1890 consensus_queue.cc:237] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.977895  1671 raft_consensus.cc:348] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.978884  1671 raft_consensus.cc:374] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.979202  1671 raft_consensus.cc:729] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.980568  1551 raft_consensus.cc:2454] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:04.982023  1442 leader_election.cc:302] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:04.982976  1743 raft_consensus.cc:2781] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:04.983455  1743 raft_consensus.cc:686] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:04.983765  1673 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:04.984254  1673 ts_tablet_manager.cc:1367] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.060s	user 0.009s	sys 0.004s
I20240426 13:23:04.984331  1743 consensus_queue.cc:237] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:04.986351  1673 raft_consensus.cc:348] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.987020  1673 raft_consensus.cc:374] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:04.987674  1673 raft_consensus.cc:729] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:04.988306  1673 consensus_queue.cc:260] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:04.988982  1671 consensus_queue.cc:260] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:04.990079  1673 ts_tablet_manager.cc:1398] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:23:04.991092  1673 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:04.991809  1671 ts_tablet_manager.cc:1398] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.050s	user 0.004s	sys 0.003s
I20240426 13:23:04.993062  1671 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:04.996790  1673 tablet_bootstrap.cc:654] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:04.999179  1671 tablet_bootstrap.cc:654] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.008014  1673 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.008498  1673 ts_tablet_manager.cc:1367] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.007s
I20240426 13:23:05.010329  1672 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.010828  1672 ts_tablet_manager.cc:1367] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.117s	user 0.023s	sys 0.026s
I20240426 13:23:05.013327  1672 raft_consensus.cc:348] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.014003  1672 raft_consensus.cc:374] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.014259  1672 raft_consensus.cc:729] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.014811  1672 consensus_queue.cc:260] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.015095  1721 consensus_queue.cc:1035] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.016957  1672 ts_tablet_manager.cc:1398] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240426 13:23:05.019321  1673 raft_consensus.cc:348] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.020061  1673 raft_consensus.cc:374] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.020303  1673 raft_consensus.cc:729] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.020951  1673 consensus_queue.cc:260] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.022229  1735 raft_consensus.cc:482] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.022920  1735 raft_consensus.cc:504] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.026610  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.027448  1400 raft_consensus.cc:2454] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:05.029031  1589 leader_election.cc:302] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:05.030164  1733 raft_consensus.cc:2781] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.030635  1733 raft_consensus.cc:482] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.031061  1733 raft_consensus.cc:3037] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.033686  1671 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.034173  1674 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.034265  1671 ts_tablet_manager.cc:1367] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.041s	user 0.017s	sys 0.000s
I20240426 13:23:05.034677  1674 ts_tablet_manager.cc:1367] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.143s	user 0.019s	sys 0.034s
I20240426 13:23:05.037448  1671 raft_consensus.cc:348] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.038391  1671 raft_consensus.cc:374] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.038664  1671 raft_consensus.cc:729] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.039489  1671 consensus_queue.cc:260] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.041924  1735 leader_election.cc:288] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.044183  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.057458  1477 raft_consensus.cc:2454] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:05.058409  1672 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.044905  1824 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.059386  1824 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.061832  1824 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.063421  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.064097  1478 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:05.064425  1672 tablet_bootstrap.cc:654] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.049636  1674 raft_consensus.cc:348] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.065418  1674 raft_consensus.cc:374] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.065701  1674 raft_consensus.cc:729] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.066812  1674 consensus_queue.cc:260] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.068915  1674 ts_tablet_manager.cc:1398] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.000s
I20240426 13:23:05.069938  1674 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.072826  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.073483  1551 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:05.051443  1671 ts_tablet_manager.cc:1398] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20240426 13:23:05.052723  1827 consensus_queue.cc:1035] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.051528  1673 ts_tablet_manager.cc:1398] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.043s	user 0.018s	sys 0.016s
I20240426 13:23:05.077327  1673 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.065677  1366 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.080790  1671 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.084264  1828 raft_consensus.cc:2781] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.084838  1828 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.085250  1828 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.088080  1674 tablet_bootstrap.cc:654] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.092427  1777 raft_consensus.cc:482] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.092953  1777 raft_consensus.cc:504] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.056536  1733 raft_consensus.cc:504] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.096074  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:05.099541  1366 leader_election.cc:341] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.101598  1735 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.102286  1735 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.108244  1777 leader_election.cc:288] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.109036  1672 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.110298  1673 tablet_bootstrap.cc:654] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.116393  1672 ts_tablet_manager.cc:1367] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.058s	user 0.010s	sys 0.001s
I20240426 13:23:05.110507  1735 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.112299  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.118549  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.113710  1671 tablet_bootstrap.cc:654] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.119102  1477 raft_consensus.cc:2379] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in this term.
I20240426 13:23:05.119076  1672 raft_consensus.cc:348] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.113795  1743 raft_consensus.cc:482] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.118098  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.119908  1672 raft_consensus.cc:374] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.122125  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7cffa351e841f2bbf2789cc31939cc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:05.120499  1733 leader_election.cc:288] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.127574  1298 catalog_manager.cc:5484] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
W20240426 13:23:05.129559  1367 leader_election.cc:341] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.129355  1298 catalog_manager.cc:5484] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.130021  1367 leader_election.cc:302] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:05.126793  1743 raft_consensus.cc:504] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.127053  1672 raft_consensus.cc:729] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.132364  1672 consensus_queue.cc:260] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.133358  1674 tablet_bootstrap.cc:492] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.133219  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95477b67a3084e64a196d9d64957a3f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:05.134452  1674 ts_tablet_manager.cc:1367] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.065s	user 0.016s	sys 0.003s
I20240426 13:23:05.134510  1626 raft_consensus.cc:2454] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.119836  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.133788  1703 raft_consensus.cc:2726] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:05.123694  1777 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.135967  1443 leader_election.cc:302] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:05.136418  1673 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.136296  1777 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.136920  1673 ts_tablet_manager.cc:1367] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.006s
I20240426 13:23:05.137120  1919 raft_consensus.cc:2781] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.137621  1919 raft_consensus.cc:482] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.137957  1919 raft_consensus.cc:3037] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.139335  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.140115  1478 raft_consensus.cc:2454] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:05.141458  1366 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.142292  1703 raft_consensus.cc:2781] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.142769  1777 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.142700  1703 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.142427  1671 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.143677  1703 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.143981  1671 ts_tablet_manager.cc:1367] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.063s	user 0.011s	sys 0.006s
I20240426 13:23:05.146729  1671 raft_consensus.cc:348] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.147349  1671 raft_consensus.cc:374] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.147607  1671 raft_consensus.cc:729] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.148218  1674 raft_consensus.cc:348] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.148891  1674 raft_consensus.cc:374] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.149179  1674 raft_consensus.cc:729] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.149819  1674 consensus_queue.cc:260] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.151870  1674 ts_tablet_manager.cc:1398] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.017s	user 0.002s	sys 0.001s
I20240426 13:23:05.152235  1589 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:05.152951  1674 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.153510  1733 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.156286  1671 consensus_queue.cc:260] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.139451  1673 raft_consensus.cc:348] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.157313  1673 raft_consensus.cc:374] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.157586  1673 raft_consensus.cc:729] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.158231  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.159093  1825 consensus_queue.cc:1035] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.159845  1673 consensus_queue.cc:260] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
W20240426 13:23:05.160681  1367 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.162577  1673 ts_tablet_manager.cc:1398] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20240426 13:23:05.168254  1673 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.171664  1672 ts_tablet_manager.cc:1398] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.055s	user 0.006s	sys 0.001s
I20240426 13:23:05.170701  1828 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.172729  1672 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.174520  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:05.175130  1475 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.119115  1403 raft_consensus.cc:3037] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.128142  1477 raft_consensus.cc:3037] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.159454  1671 ts_tablet_manager.cc:1398] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.001s
I20240426 13:23:05.176725  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95477b67a3084e64a196d9d64957a3f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.165656  1674 tablet_bootstrap.cc:654] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.187309  1402 raft_consensus.cc:2454] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.166280  1703 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.189258  1673 tablet_bootstrap.cc:654] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.182137  1475 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:05.190850  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:05.191468  1476 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.187358  1671 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.200498  1476 raft_consensus.cc:2454] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:05.164960  1733 raft_consensus.cc:2726] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:05.166939  1743 leader_election.cc:288] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.188302  1672 tablet_bootstrap.cc:654] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.203881  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:05.204602  1551 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.206769  1703 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.201282  1366 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.207430  1777 consensus_queue.cc:1035] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.207935  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:05.211851  1825 raft_consensus.cc:2781] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:23:05.212726  1367 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.201566  1828 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.213024  1825 raft_consensus.cc:686] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:05.211254  1674 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.213829  1674 ts_tablet_manager.cc:1367] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.061s	user 0.004s	sys 0.008s
I20240426 13:23:05.214054  1825 consensus_queue.cc:237] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.216288  1366 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:05.216867  1674 raft_consensus.cc:348] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.217746  1674 raft_consensus.cc:374] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.218062  1674 raft_consensus.cc:729] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.200929  1671 tablet_bootstrap.cc:654] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.218765  1824 raft_consensus.cc:2781] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.219388  1824 raft_consensus.cc:686] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:05.210106  1403 raft_consensus.cc:2454] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:05.220443  1824 consensus_queue.cc:237] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.221482  1589 leader_election.cc:302] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:05.223754  1735 raft_consensus.cc:2781] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.228627  1680 consensus_queue.cc:1035] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.233059  1672 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.232749  1919 raft_consensus.cc:504] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.233574  1672 ts_tablet_manager.cc:1367] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.061s	user 0.007s	sys 0.005s
I20240426 13:23:05.201031  1477 raft_consensus.cc:2454] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:05.238816  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95477b67a3084e64a196d9d64957a3f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:05.239576  1622 raft_consensus.cc:3037] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.244019  1551 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:05.244509  1735 raft_consensus.cc:686] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:05.244920  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95477b67a3084e64a196d9d64957a3f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.245538  1400 raft_consensus.cc:3037] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.245421  1735 consensus_queue.cc:237] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.252585  1672 raft_consensus.cc:348] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.253314  1672 raft_consensus.cc:374] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.253595  1672 raft_consensus.cc:729] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.254098  1671 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.256141  1400 raft_consensus.cc:2454] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:05.257230  1672 consensus_queue.cc:260] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.262905  1743 raft_consensus.cc:482] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.277225  1672 ts_tablet_manager.cc:1398] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.043s	user 0.003s	sys 0.004s
I20240426 13:23:05.277264  1743 raft_consensus.cc:504] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.278478  1672 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.263638  1680 consensus_queue.cc:1035] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.281052  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:05.284147  1443 leader_election.cc:341] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.271750  1297 catalog_manager.cc:5484] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.273504  1673 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.274616  1890 raft_consensus.cc:482] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.291491  1890 raft_consensus.cc:504] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.293624  1890 leader_election.cc:288] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.277737  1674 consensus_queue.cc:260] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.294427  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3bd4a614b14384a7d4945594ab3871" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:05.295564  1626 raft_consensus.cc:2454] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.238840  1919 leader_election.cc:288] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.294952  1673 ts_tablet_manager.cc:1367] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.127s	user 0.014s	sys 0.000s
I20240426 13:23:05.271672  1622 raft_consensus.cc:2454] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:05.282255  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.298506  1443 leader_election.cc:302] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:05.298688  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3bd4a614b14384a7d4945594ab3871" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.297331  1441 leader_election.cc:302] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.254586  1671 ts_tablet_manager.cc:1367] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.067s	user 0.010s	sys 0.002s
I20240426 13:23:05.299631  1400 raft_consensus.cc:2454] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.300165  1890 raft_consensus.cc:2781] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.300573  1890 raft_consensus.cc:482] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.300904  1890 raft_consensus.cc:3037] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.301604  1671 raft_consensus.cc:348] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.284956  1743 leader_election.cc:288] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:05.295733  1674 ts_tablet_manager.cc:1398] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.082s	user 0.005s	sys 0.001s
I20240426 13:23:05.298993  1551 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.302341  1671 raft_consensus.cc:374] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.303217  1671 raft_consensus.cc:729] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.303921  1671 consensus_queue.cc:260] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.304720  1673 raft_consensus.cc:348] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.305439  1673 raft_consensus.cc:374] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.305557  1674 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.305961  1442 leader_election.cc:302] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:05.307791  1743 raft_consensus.cc:2781] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.305739  1673 raft_consensus.cc:729] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.308283  1743 raft_consensus.cc:686] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:05.308408  1682 raft_consensus.cc:2781] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.297451  1672 tablet_bootstrap.cc:654] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.320374  1673 consensus_queue.cc:260] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.320385  1682 raft_consensus.cc:482] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.320763  1743 consensus_queue.cc:237] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.321444  1682 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.312855  1671 ts_tablet_manager.cc:1398] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20240426 13:23:05.333709  1671 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.310530  1890 raft_consensus.cc:504] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.319201  1301 catalog_manager.cc:5484] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.338682  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3bd4a614b14384a7d4945594ab3871" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:05.339274  1626 raft_consensus.cc:3037] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.342041  1674 tablet_bootstrap.cc:654] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.352560  1298 catalog_manager.cc:5484] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.344992  1671 tablet_bootstrap.cc:654] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.348430  1826 raft_consensus.cc:482] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.354039  1826 raft_consensus.cc:504] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.356549  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:05.357841  1517 leader_election.cc:341] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.358737  1826 leader_election.cc:288] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.348150  1672 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.359158  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3bd4a614b14384a7d4945594ab3871" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.360011  1402 raft_consensus.cc:3037] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.359539  1672 ts_tablet_manager.cc:1367] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.081s	user 0.012s	sys 0.000s
I20240426 13:23:05.362659  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.363312  1400 raft_consensus.cc:2454] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:05.364545  1515 leader_election.cc:302] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:05.366055  1826 raft_consensus.cc:2781] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.366417  1826 raft_consensus.cc:482] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.366787  1826 raft_consensus.cc:3037] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.364583  1672 raft_consensus.cc:348] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.371831  1672 raft_consensus.cc:374] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.372191  1672 raft_consensus.cc:729] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.372900  1672 consensus_queue.cc:260] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.345650  1682 raft_consensus.cc:504] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.375072  1672 ts_tablet_manager.cc:1398] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.002s
I20240426 13:23:05.378855  1672 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.380542  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:05.381230  1549 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.386082  1672 tablet_bootstrap.cc:654] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.388490  1919 consensus_queue.cc:1035] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.393884  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:05.393981  1682 leader_election.cc:288] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:05.392313  1890 leader_election.cc:288] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
W20240426 13:23:05.408052  1443 leader_election.cc:341] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.392002  1402 raft_consensus.cc:2454] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:05.410672  1441 leader_election.cc:302] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.412096  1890 raft_consensus.cc:2781] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.413062  1890 raft_consensus.cc:686] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:05.414129  1549 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:05.415469  1442 leader_election.cc:302] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:05.417387  1682 raft_consensus.cc:2781] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.417989  1682 raft_consensus.cc:686] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:05.414410  1890 consensus_queue.cc:237] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.421063  1673 ts_tablet_manager.cc:1398] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.123s	user 0.015s	sys 0.058s
I20240426 13:23:05.420923  1626 raft_consensus.cc:2454] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:05.423177  1671 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.423871  1671 ts_tablet_manager.cc:1367] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.090s	user 0.010s	sys 0.000s
I20240426 13:23:05.422206  1673 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.431160  1674 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.419227  1682 consensus_queue.cc:237] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.436759  1674 ts_tablet_manager.cc:1367] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.131s	user 0.010s	sys 0.002s
I20240426 13:23:05.438164  1297 catalog_manager.cc:5484] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:05.432487  1673 tablet_bootstrap.cc:654] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.439270  1674 raft_consensus.cc:348] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.439728  1624 raft_consensus.cc:3037] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.428768  1826 raft_consensus.cc:504] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.444248  1721 raft_consensus.cc:482] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.444725  1721 raft_consensus.cc:504] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.446664  1721 leader_election.cc:288] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.448550  1826 leader_election.cc:288] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.435454  1671 raft_consensus.cc:348] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.450893  1671 raft_consensus.cc:374] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.451013  1674 raft_consensus.cc:374] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.451119  1671 raft_consensus.cc:729] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.451568  1674 raft_consensus.cc:729] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.450244  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.452384  1674 consensus_queue.cc:260] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.453797  1671 consensus_queue.cc:260] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.453982  1938 consensus_queue.cc:1035] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.457588  1671 ts_tablet_manager.cc:1398] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20240426 13:23:05.457640  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.458361  1400 raft_consensus.cc:3037] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.458958  1674 ts_tablet_manager.cc:1398] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20240426 13:23:05.459287  1671 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.460068  1674 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.460427  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.461086  1477 raft_consensus.cc:2454] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:05.461722  1402 raft_consensus.cc:2454] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:05.463541  1515 leader_election.cc:302] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:05.464344  1721 raft_consensus.cc:2781] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.464705  1721 raft_consensus.cc:482] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.464996  1721 raft_consensus.cc:3037] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.466161  1674 tablet_bootstrap.cc:654] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.467294  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a02c7dd614ad8b152d809ebd56155" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:05.467965  1626 raft_consensus.cc:3037] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.472525  1721 raft_consensus.cc:504] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.471052  1880 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.473834  1880 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.474516  1721 leader_election.cc:288] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.475746  1880 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:05.476385  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.476958  1402 raft_consensus.cc:3037] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.481104  1938 consensus_queue.cc:1035] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.483261  1402 raft_consensus.cc:2454] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.487038  1672 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.487265  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ce3aa126ef422ca8fedc1e1349087b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:05.488090  1476 raft_consensus.cc:3037] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.487757  1672 ts_tablet_manager.cc:1367] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.110s	user 0.015s	sys 0.019s
I20240426 13:23:05.494689  1672 raft_consensus.cc:348] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.495600  1672 raft_consensus.cc:374] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.497110  1515 leader_election.cc:302] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:05.498001  1721 raft_consensus.cc:2781] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.499322  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.499056  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.500471  1477 raft_consensus.cc:2379] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in this term.
I20240426 13:23:05.499760  1721 raft_consensus.cc:686] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:05.501443  1721 consensus_queue.cc:237] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.499608  1400 raft_consensus.cc:2454] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.495918  1672 raft_consensus.cc:729] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.511930  1671 tablet_bootstrap.cc:654] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.509621  1672 consensus_queue.cc:260] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.511330  1476 raft_consensus.cc:2454] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.526027  1672 ts_tablet_manager.cc:1398] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.035s	user 0.009s	sys 0.000s
I20240426 13:23:05.521287  1626 raft_consensus.cc:2454] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.527395  1515 leader_election.cc:302] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:05.528522  1673 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.529596  1880 raft_consensus.cc:482] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.535827  1673 ts_tablet_manager.cc:1367] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.114s	user 0.017s	sys 0.026s
I20240426 13:23:05.539521  1880 raft_consensus.cc:504] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.554780  1880 leader_election.cc:288] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.554965  1515 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:05.554924  1299 catalog_manager.cc:5484] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:05.566294  1673 raft_consensus.cc:348] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.544884  1721 raft_consensus.cc:2781] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.570278  1673 raft_consensus.cc:374] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.546496  1672 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.570793  1673 raft_consensus.cc:729] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.574056  1673 consensus_queue.cc:260] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.576644  1673 ts_tablet_manager.cc:1398] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20240426 13:23:05.578213  1673 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.582234  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.586752  1402 raft_consensus.cc:2454] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:05.587234  1890 consensus_queue.cc:1035] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.589566  1673 tablet_bootstrap.cc:654] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.570778  1880 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.556183  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:05.572489  1721 raft_consensus.cc:686] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:05.597340  1674 tablet_bootstrap.cc:492] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.601372  1674 ts_tablet_manager.cc:1367] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.141s	user 0.023s	sys 0.018s
W20240426 13:23:05.597476  1517 leader_election.cc:341] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.607086  1297 catalog_manager.cc:5484] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:05.608151  1674 raft_consensus.cc:348] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.609067  1674 raft_consensus.cc:374] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.610406  1674 raft_consensus.cc:729] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.611855  1674 consensus_queue.cc:260] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.609052  1297 catalog_manager.cc:5484] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.616211  1672 tablet_bootstrap.cc:654] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.621034  1735 raft_consensus.cc:482] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.611271  1721 consensus_queue.cc:237] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.621626  1735 raft_consensus.cc:504] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.624384  1735 leader_election.cc:288] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.625082  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.606166  1515 leader_election.cc:302] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:05.622349  1674 ts_tablet_manager.cc:1398] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20240426 13:23:05.628855  1674 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.629458  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.630513  1551 raft_consensus.cc:2379] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:05.630709  1880 raft_consensus.cc:2726] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:05.631659  1827 raft_consensus.cc:2781] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.632223  1827 raft_consensus.cc:482] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.632566  1827 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.633208  1590 leader_election.cc:302] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:05.633925  1671 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.634675  1671 ts_tablet_manager.cc:1367] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.176s	user 0.018s	sys 0.014s
I20240426 13:23:05.635043  1934 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.637387  1828 raft_consensus.cc:482] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.637960  1828 raft_consensus.cc:504] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.640144  1671 raft_consensus.cc:348] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.640831  1671 raft_consensus.cc:374] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.641108  1671 raft_consensus.cc:729] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.641469  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.642095  1934 raft_consensus.cc:2726] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:23:05.642937  1367 leader_election.cc:341] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.643584  1674 tablet_bootstrap.cc:654] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.643898  1828 leader_election.cc:288] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:05.644665  1827 raft_consensus.cc:504] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.660612  1827 leader_election.cc:288] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:05.661598  1674 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.661557  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:05.662210  1674 ts_tablet_manager.cc:1367] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.034s	user 0.007s	sys 0.004s
I20240426 13:23:05.662215  1400 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.665177  1674 raft_consensus.cc:348] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.665879  1674 raft_consensus.cc:374] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.666113  1674 raft_consensus.cc:729] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.666685  1674 consensus_queue.cc:260] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.668772  1674 ts_tablet_manager.cc:1398] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:23:05.668745  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:05.669494  1626 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.669687  1400 raft_consensus.cc:2454] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.669946  1674 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.670707  1515 leader_election.cc:302] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:05.671442  1827 raft_consensus.cc:2781] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:05.671824  1827 raft_consensus.cc:686] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:05.672956  1827 consensus_queue.cc:237] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.673600  1934 consensus_queue.cc:1035] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.675906  1674 tablet_bootstrap.cc:654] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.644948  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:05.649827  1671 consensus_queue.cc:260] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.653247  1828 consensus_queue.cc:1035] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:23:05.680012  1368 leader_election.cc:341] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.680696  1368 leader_election.cc:302] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:05.654422  1672 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.657105  1673 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.681699  1672 ts_tablet_manager.cc:1367] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.140s	user 0.013s	sys 0.000s
I20240426 13:23:05.681707  1673 ts_tablet_manager.cc:1367] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.104s	user 0.009s	sys 0.004s
I20240426 13:23:05.682163  1828 raft_consensus.cc:2726] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:05.686785  1673 raft_consensus.cc:348] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.687636  1673 raft_consensus.cc:374] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.688925  1672 raft_consensus.cc:348] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.690307  1672 raft_consensus.cc:374] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.690726  1672 raft_consensus.cc:729] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.687970  1673 raft_consensus.cc:729] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.692205  1673 consensus_queue.cc:260] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.693570  1672 consensus_queue.cc:260] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.676465  1626 raft_consensus.cc:2454] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:05.699662  1673 ts_tablet_manager.cc:1398] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20240426 13:23:05.717792  1929 consensus_queue.cc:1035] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.719627  1703 consensus_queue.cc:1035] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.728556  1672 ts_tablet_manager.cc:1398] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.002s
I20240426 13:23:05.729697  1672 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.725931  1671 ts_tablet_manager.cc:1398] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.091s	user 0.006s	sys 0.000s
I20240426 13:23:05.734418  1671 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.747084  1682 consensus_queue.cc:1035] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.778434  1929 consensus_queue.cc:1035] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.782361  1721 raft_consensus.cc:482] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.782961  1721 raft_consensus.cc:504] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.741247  1297 catalog_manager.cc:5484] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.792798  1673 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.797350  1671 tablet_bootstrap.cc:654] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.807168  1890 consensus_queue.cc:1035] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:05.800426  1735 consensus_queue.cc:1035] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240426 13:23:05.820772  1674 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:05.821303  1674 ts_tablet_manager.cc:1367] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.152s	user 0.022s	sys 0.036s
I20240426 13:23:05.824424  1674 raft_consensus.cc:348] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.825150  1674 raft_consensus.cc:374] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.825426  1674 raft_consensus.cc:729] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.826059  1674 consensus_queue.cc:260] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.827380  1672 tablet_bootstrap.cc:654] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.829931  1674 ts_tablet_manager.cc:1398] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240426 13:23:05.828915  1938 raft_consensus.cc:482] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.832638  1938 raft_consensus.cc:504] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.837775  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.839465  1721 leader_election.cc:288] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:05.839676  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.840381  1402 raft_consensus.cc:2454] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:05.843618  1680 raft_consensus.cc:482] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:05.844576  1680 raft_consensus.cc:504] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.844941  1682 consensus_queue.cc:1035] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.846585  1680 leader_election.cc:288] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
W20240426 13:23:05.847304  1515 leader_election.cc:341] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.847261  1674 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:05.851992  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:05.852701  1476 raft_consensus.cc:2454] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:05.856438  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:05.857834  1517 leader_election.cc:341] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.858346  1517 leader_election.cc:302] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:05.859481  1721 raft_consensus.cc:2726] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:05.860124  1674 tablet_bootstrap.cc:654] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.862115  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:05.844978  1673 tablet_bootstrap.cc:654] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
W20240426 13:23:05.880299  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:05.885155  1589 leader_election.cc:302] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:05.923959  1680 raft_consensus.cc:2781] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.924396  1680 raft_consensus.cc:482] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.924750  1680 raft_consensus.cc:3037] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.886816  1441 leader_election.cc:302] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:05.894165  1938 leader_election.cc:288] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
W20240426 13:23:05.930716  1441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 81494.6us
I20240426 13:23:05.931586  1919 raft_consensus.cc:2781] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:05.932005  1919 raft_consensus.cc:482] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:05.932358  1919 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:05.932447  1938 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.934950  1680 raft_consensus.cc:504] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.939055  1743 consensus_queue.cc:1035] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.896921  1673 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:05.897238  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:05.899312  1672 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:05.921203  1671 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:05.950574  1672 ts_tablet_manager.cc:1367] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.221s	user 0.011s	sys 0.000s
I20240426 13:23:05.952620  1919 raft_consensus.cc:504] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:05.950609  1671 ts_tablet_manager.cc:1367] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.216s	user 0.018s	sys 0.000s
W20240426 13:23:05.953862  1442 leader_election.cc:341] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.955698  1919 leader_election.cc:288] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:05.956153  1672 raft_consensus.cc:348] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.960494  1672 raft_consensus.cc:374] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.960873  1672 raft_consensus.cc:729] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:05.956832  1671 raft_consensus.cc:348] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.958056  1673 ts_tablet_manager.cc:1367] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.169s	user 0.011s	sys 0.000s
I20240426 13:23:05.959470  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:05.961627  1671 raft_consensus.cc:374] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.961933  1671 raft_consensus.cc:729] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
W20240426 13:23:05.962774  1442 leader_election.cc:341] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.962672  1671 consensus_queue.cc:260] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:05.964738  1671 ts_tablet_manager.cc:1398] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20240426 13:23:05.965767  1671 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:05.967635  1673 raft_consensus.cc:348] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.970228  1673 raft_consensus.cc:374] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:05.970595  1673 raft_consensus.cc:729] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
W20240426 13:23:05.970844  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 45350.9us
I20240426 13:23:05.968281  1672 consensus_queue.cc:260] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:05.971431  1673 consensus_queue.cc:260] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:05.974174  1673 ts_tablet_manager.cc:1398] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240426 13:23:05.972679  1297 catalog_manager.cc:5484] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:05.975793  1673 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:05.976727  1671 tablet_bootstrap.cc:654] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
W20240426 13:23:05.971663  1589 leader_election.cc:341] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:05.973637  1672 ts_tablet_manager.cc:1398] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20240426 13:23:05.979669  1672 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:05.984853  1890 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.988797  1672 tablet_bootstrap.cc:654] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:05.990669  1827 consensus_queue.cc:1035] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:05.992745  1673 tablet_bootstrap.cc:654] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.006461  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:06.007217  1402 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.008174  1672 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:06.008692  1672 ts_tablet_manager.cc:1367] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.000s
I20240426 13:23:06.017683  1672 raft_consensus.cc:348] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.018358  1672 raft_consensus.cc:374] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.018637  1672 raft_consensus.cc:729] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.021451  1672 consensus_queue.cc:260] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.021904  1880 consensus_queue.cc:1035] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:06.020013  1673 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:06.023586  1673 ts_tablet_manager.cc:1367] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.000s
I20240426 13:23:06.024672  1703 raft_consensus.cc:482] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.025835  1672 ts_tablet_manager.cc:1398] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20240426 13:23:06.025544  1703 raft_consensus.cc:504] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.026809  1672 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:06.029179  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.030244  1626 raft_consensus.cc:2379] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e4ab2f7d204691a8f5a355a66a76a4 in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:06.032377  1703 leader_election.cc:288] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.035519  1672 tablet_bootstrap.cc:654] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.038213  1671 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.038764  1671 ts_tablet_manager.cc:1367] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.073s	user 0.016s	sys 0.019s
I20240426 13:23:06.039430  1673 raft_consensus.cc:348] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.040912  1673 raft_consensus.cc:374] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.041534  1673 raft_consensus.cc:729] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.041505  1671 raft_consensus.cc:348] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.041589  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.043233  1478 raft_consensus.cc:2454] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:06.044201  1671 raft_consensus.cc:374] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.044652  1671 raft_consensus.cc:729] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.044688  1366 leader_election.cc:302] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.043874  1673 consensus_queue.cc:260] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.045656  1671 consensus_queue.cc:260] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.046159  1703 raft_consensus.cc:2781] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.046694  1703 raft_consensus.cc:482] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.047086  1703 raft_consensus.cc:3037] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.048022  1673 ts_tablet_manager.cc:1398] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20240426 13:23:06.048019  1671 ts_tablet_manager.cc:1398] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20240426 13:23:06.052460  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:06.058708  1671 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.067509  1673 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:06.067972  1671 tablet_bootstrap.cc:654] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.068943  1721 consensus_queue.cc:1035] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:06.070783  1674 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:06.071270  1674 ts_tablet_manager.cc:1367] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.224s	user 0.026s	sys 0.042s
I20240426 13:23:06.074263  1673 tablet_bootstrap.cc:654] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.079403  1674 raft_consensus.cc:348] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.080201  1674 raft_consensus.cc:374] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.080518  1674 raft_consensus.cc:729] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.081605  1671 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.082091  1671 ts_tablet_manager.cc:1367] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.003s
I20240426 13:23:06.083601  1674 consensus_queue.cc:260] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.086882  1674 ts_tablet_manager.cc:1398] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20240426 13:23:06.100258  1674 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:06.104976  1880 consensus_queue.cc:1035] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:06.106601  1674 tablet_bootstrap.cc:654] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.113735  1826 raft_consensus.cc:482] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.114272  1826 raft_consensus.cc:504] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.053958  1680 leader_election.cc:288] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.054425  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:06.054304  1703 raft_consensus.cc:504] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.120211  1703 leader_election.cc:288] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.120321  1671 raft_consensus.cc:348] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.121105  1671 raft_consensus.cc:374] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.121359  1671 raft_consensus.cc:729] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.121986  1671 consensus_queue.cc:260] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.125844  1477 raft_consensus.cc:3037] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.126371  1674 tablet_bootstrap.cc:492] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:06.126892  1674 ts_tablet_manager.cc:1367] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.001s
I20240426 13:23:06.129146  1671 ts_tablet_manager.cc:1398] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.000s
I20240426 13:23:06.129647  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:06.129979  1880 consensus_queue.cc:1035] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:06.130172  1826 leader_election.cc:288] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.130182  1671 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.134019  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.134863  1478 raft_consensus.cc:2379] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 1: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:06.136655  1674 raft_consensus.cc:348] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.137627  1674 raft_consensus.cc:374] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.137964  1674 raft_consensus.cc:729] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.139169  1671 tablet_bootstrap.cc:654] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.143755  1674 consensus_queue.cc:260] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.148524  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e00f8df835646e89148ac3e3a2d6abc" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:06.156455  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:06.162111  1399 raft_consensus.cc:2379] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in this term.
I20240426 13:23:06.162910  1880 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.130532  1622 raft_consensus.cc:2379] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 56e4ab2f7d204691a8f5a355a66a76a4 in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:06.163479  1880 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.164729  1368 leader_election.cc:302] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.165314  1671 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.165807  1671 ts_tablet_manager.cc:1367] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.008s
I20240426 13:23:06.166277  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.166911  1703 raft_consensus.cc:2726] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240426 13:23:06.167635  1517 leader_election.cc:341] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.168563  1880 leader_election.cc:288] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:06.171609  1674 ts_tablet_manager.cc:1398] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20240426 13:23:06.172710  1674 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:06.168342  1671 raft_consensus.cc:348] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.173622  1671 raft_consensus.cc:374] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.173908  1671 raft_consensus.cc:729] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.175455  1671 consensus_queue.cc:260] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.179423  1402 raft_consensus.cc:2454] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:06.179956  1671 ts_tablet_manager.cc:1398] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20240426 13:23:06.182096  1671 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.181932  1441 leader_election.cc:302] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.182986  1743 raft_consensus.cc:2781] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.183529  1743 raft_consensus.cc:686] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:06.184361  1743 consensus_queue.cc:237] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.189867  1671 tablet_bootstrap.cc:654] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.190850  1477 raft_consensus.cc:2454] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:06.211642  1703 raft_consensus.cc:482] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.212247  1703 raft_consensus.cc:504] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.212692  1589 leader_election.cc:302] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:06.214186  1680 raft_consensus.cc:2781] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.214638  1680 raft_consensus.cc:686] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:06.215410  1680 consensus_queue.cc:237] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.216037  1673 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:06.216632  1673 ts_tablet_manager.cc:1367] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.149s	user 0.019s	sys 0.010s
I20240426 13:23:06.219214  1673 raft_consensus.cc:348] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.219877  1703 leader_election.cc:288] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.222730  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.222981  1671 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.223467  1671 ts_tablet_manager.cc:1367] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.042s	user 0.020s	sys 0.010s
I20240426 13:23:06.223874  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:06.220714  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.224920  1626 raft_consensus.cc:2454] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:06.200803  1672 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:06.203692  1674 tablet_bootstrap.cc:654] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.219949  1673 raft_consensus.cc:374] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.226020  1368 leader_election.cc:302] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:06.226549  1673 raft_consensus.cc:729] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.227322  1673 consensus_queue.cc:260] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.209908  1515 leader_election.cc:302] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
W20240426 13:23:06.227766  1366 leader_election.cc:341] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:06.229384  1515 leader_election.cc:341] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.228668  1827 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.236250  1672 ts_tablet_manager.cc:1367] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.210s	user 0.022s	sys 0.014s
I20240426 13:23:06.238682  1672 raft_consensus.cc:348] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.239305  1672 raft_consensus.cc:374] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.239537  1672 raft_consensus.cc:729] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.230343  1721 consensus_queue.cc:1035] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:06.228658  1703 raft_consensus.cc:2781] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.240965  1703 raft_consensus.cc:482] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.241333  1703 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.235782  1515 leader_election.cc:302] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.245332  1721 raft_consensus.cc:2726] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.251149  1827 raft_consensus.cc:2726] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.225910  1671 raft_consensus.cc:348] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.256667  1671 raft_consensus.cc:374] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.256947  1671 raft_consensus.cc:729] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.258253  1671 consensus_queue.cc:260] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.260864  1673 ts_tablet_manager.cc:1398] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20240426 13:23:06.261844  1673 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:06.263640  1298 catalog_manager.cc:5484] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:06.265736  1671 ts_tablet_manager.cc:1398] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.042s	user 0.002s	sys 0.005s
I20240426 13:23:06.271142  1703 raft_consensus.cc:504] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.273799  1672 consensus_queue.cc:260] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.273995  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:06.274637  1626 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.275568  1672 ts_tablet_manager.cc:1398] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.000s
I20240426 13:23:06.276484  1672 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:06.281464  1626 raft_consensus.cc:2454] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:06.282346  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:06.282603  1368 leader_election.cc:302] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:06.283509  1828 raft_consensus.cc:2781] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.283983  1828 raft_consensus.cc:686] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
W20240426 13:23:06.284243  1366 leader_election.cc:341] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.284773  1828 consensus_queue.cc:237] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.286284  1703 leader_election.cc:288] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.285768  1671 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.288601  1672 tablet_bootstrap.cc:654] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.294132  1671 tablet_bootstrap.cc:654] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.297377  1673 tablet_bootstrap.cc:654] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.295184  1301 catalog_manager.cc:5484] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:06.302309  1929 raft_consensus.cc:482] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.302807  1929 raft_consensus.cc:504] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.305409  1929 leader_election.cc:288] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.317976  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.320866  1674 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:06.321646  1674 ts_tablet_manager.cc:1367] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.149s	user 0.009s	sys 0.030s
I20240426 13:23:06.321767  1994 raft_consensus.cc:482] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.322368  1994 raft_consensus.cc:504] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.324671  1994 leader_election.cc:288] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:06.324878  1674 raft_consensus.cc:348] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.325793  1674 raft_consensus.cc:374] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.326112  1674 raft_consensus.cc:729] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
W20240426 13:23:06.326172  1368 leader_election.cc:341] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.326404  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.326969  1674 consensus_queue.cc:260] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
W20240426 13:23:06.328905  1517 leader_election.cc:341] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.329763  1674 ts_tablet_manager.cc:1398] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240426 13:23:06.331180  1743 raft_consensus.cc:482] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.331689  1743 raft_consensus.cc:504] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.333720  1743 leader_election.cc:288] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.333787  1674 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:06.337622  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:06.337718  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:06.338516  1403 raft_consensus.cc:2454] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
W20240426 13:23:06.338893  1442 leader_election.cc:341] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.339668  1929 raft_consensus.cc:482] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.340178  1441 leader_election.cc:302] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.341756  1890 raft_consensus.cc:2781] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.342173  1890 raft_consensus.cc:482] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.342545  1890 raft_consensus.cc:3037] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.340121  1929 raft_consensus.cc:504] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.344357  1929 leader_election.cc:288] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.347364  1674 tablet_bootstrap.cc:654] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.349821  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.352649  1890 raft_consensus.cc:504] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.354231  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.354882  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:06.367722  1367 leader_election.cc:341] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.370448  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:06.371270  1403 raft_consensus.cc:3037] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.376811  1476 raft_consensus.cc:2379] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e4ab2f7d204691a8f5a355a66a76a4 in current term 1: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:06.378623  1735 raft_consensus.cc:482] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.379109  1735 raft_consensus.cc:504] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.380972  1735 leader_election.cc:288] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.383222  1735 raft_consensus.cc:482] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.383677  1735 raft_consensus.cc:504] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.385556  1735 leader_election.cc:288] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.385763  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:06.386833  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:06.387503  1548 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
W20240426 13:23:06.387660  1442 leader_election.cc:341] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.388607  1590 leader_election.cc:302] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:06.389616  1890 leader_election.cc:288] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.390728  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.391880  1477 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:06.391476  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7a62fac3114417ae31d5ce8f6bbd86" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.393113  1476 raft_consensus.cc:2454] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:06.394896  1589 leader_election.cc:302] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:06.395700  1680 raft_consensus.cc:2781] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.395859  1735 raft_consensus.cc:2781] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.396219  1680 raft_consensus.cc:482] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.396737  1680 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.397050  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7a62fac3114417ae31d5ce8f6bbd86" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:06.398327  1590 leader_election.cc:341] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.398650  1671 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.396394  1735 raft_consensus.cc:482] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.399125  1671 ts_tablet_manager.cc:1367] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.133s	user 0.011s	sys 0.019s
I20240426 13:23:06.399359  1735 raft_consensus.cc:3037] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.401310  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:06.401686  1366 leader_election.cc:341] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.401957  1399 raft_consensus.cc:2454] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:06.402164  1366 leader_election.cc:302] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.383214  1403 raft_consensus.cc:2454] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:06.406709  1441 leader_election.cc:302] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.407766  1680 raft_consensus.cc:504] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.409245  1938 raft_consensus.cc:2781] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.409766  1938 raft_consensus.cc:686] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:06.410297  1671 raft_consensus.cc:348] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.410962  1671 raft_consensus.cc:374] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.410607  1938 consensus_queue.cc:237] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.411255  1671 raft_consensus.cc:729] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.411283  1680 leader_election.cc:288] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.411886  1671 consensus_queue.cc:260] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.414724  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:06.416510  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:06.419257  1549 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.415823  1476 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.436568  1929 raft_consensus.cc:2726] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.450734  1300 catalog_manager.cc:5484] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:06.403714  1366 leader_election.cc:302] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.453822  1828 raft_consensus.cc:2726] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:23:06.472146  1366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 68487.1us
I20240426 13:23:06.474439  1515 leader_election.cc:302] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.475749  1994 raft_consensus.cc:2781] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.476154  1994 raft_consensus.cc:482] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.476469  1994 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.483081  1672 tablet_bootstrap.cc:492] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:06.483606  1672 ts_tablet_manager.cc:1367] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.207s	user 0.029s	sys 0.022s
I20240426 13:23:06.484670  1735 raft_consensus.cc:504] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.486385  1672 raft_consensus.cc:348] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.488018  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7a62fac3114417ae31d5ce8f6bbd86" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:06.488605  1671 ts_tablet_manager.cc:1398] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.089s	user 0.041s	sys 0.024s
I20240426 13:23:06.488700  1477 raft_consensus.cc:3037] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.489791  1671 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.490723  1994 raft_consensus.cc:504] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.493566  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
W20240426 13:23:06.494212  1368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 60861.4us
W20240426 13:23:06.494902  1517 leader_election.cc:341] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.495674  1477 raft_consensus.cc:2454] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:06.496852  1589 leader_election.cc:302] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:06.497620  1673 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:06.497704  1973 raft_consensus.cc:2781] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.493126  1994 leader_election.cc:288] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:06.498317  1973 raft_consensus.cc:686] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:06.499105  1300 catalog_manager.cc:5484] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:06.499913  1973 consensus_queue.cc:237] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.487112  1672 raft_consensus.cc:374] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.502338  1674 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:06.502450  1672 raft_consensus.cc:729] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.502921  1674 ts_tablet_manager.cc:1367] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.169s	user 0.013s	sys 0.024s
I20240426 13:23:06.503515  1672 consensus_queue.cc:260] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.498483  2000 raft_consensus.cc:482] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.498335  1673 ts_tablet_manager.cc:1367] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.237s	user 0.027s	sys 0.021s
I20240426 13:23:06.506166  1672 ts_tablet_manager.cc:1398] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20240426 13:23:06.508009  2000 raft_consensus.cc:504] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.508843  1673 raft_consensus.cc:348] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.509495  1476 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:06.509718  1673 raft_consensus.cc:374] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.510082  1673 raft_consensus.cc:729] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.510744  1673 consensus_queue.cc:260] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.512260  2000 leader_election.cc:288] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:06.505618  1674 raft_consensus.cc:348] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.513329  1674 raft_consensus.cc:374] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.513866  1674 raft_consensus.cc:729] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.517087  1673 ts_tablet_manager.cc:1398] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240426 13:23:06.518163  1673 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:06.520180  1671 tablet_bootstrap.cc:654] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.521093  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.522943  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:06.522945  1368 leader_election.cc:341] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:06.524922  1367 leader_election.cc:341] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.525341  1367 leader_election.cc:302] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.525836  1549 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:06.526954  1590 leader_election.cc:302] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:06.527849  1934 raft_consensus.cc:2781] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.528088  2002 raft_consensus.cc:2726] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.528450  1934 raft_consensus.cc:686] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:06.529353  1934 consensus_queue.cc:237] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.530230  1672 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:06.514762  1674 consensus_queue.cc:260] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.535617  1674 ts_tablet_manager.cc:1398] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.002s
I20240426 13:23:06.536007  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:06.536679  1399 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.536844  1674 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:06.542995  1672 tablet_bootstrap.cc:654] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.544127  1399 raft_consensus.cc:2454] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:06.545691  1515 leader_election.cc:302] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.546509  1994 raft_consensus.cc:2781] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.546977  1994 raft_consensus.cc:686] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:06.547263  1671 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.547793  1671 ts_tablet_manager.cc:1367] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.058s	user 0.009s	sys 0.000s
I20240426 13:23:06.548022  1994 consensus_queue.cc:237] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.550315  1671 raft_consensus.cc:348] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.551075  1671 raft_consensus.cc:374] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.551337  1671 raft_consensus.cc:729] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.551617  1674 tablet_bootstrap.cc:654] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.551936  1671 consensus_queue.cc:260] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.555647  1671 ts_tablet_manager.cc:1398] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240426 13:23:06.574930  1673 tablet_bootstrap.cc:654] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.575137  1298 catalog_manager.cc:5484] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:06.576169  1671 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.570287  1735 leader_election.cc:288] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.570482  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7a62fac3114417ae31d5ce8f6bbd86" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:06.594900  1549 raft_consensus.cc:3037] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.611821  1672 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:06.612588  1672 ts_tablet_manager.cc:1367] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.083s	user 0.020s	sys 0.015s
I20240426 13:23:06.615309  1671 tablet_bootstrap.cc:654] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.620065  1674 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:06.620716  1674 ts_tablet_manager.cc:1367] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.084s	user 0.021s	sys 0.009s
I20240426 13:23:06.621338  1298 catalog_manager.cc:5484] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:06.629912  1828 raft_consensus.cc:482] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.632651  1828 raft_consensus.cc:504] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.635202  1828 leader_election.cc:288] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.641549  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:06.644707  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.648291  1938 consensus_queue.cc:1035] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:23:06.658747  1368 leader_election.cc:341] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:06.659946  1366 leader_election.cc:341] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.660565  1366 leader_election.cc:302] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:06.662753  2004 raft_consensus.cc:2726] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.689067  1672 raft_consensus.cc:348] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.662590  1549 raft_consensus.cc:2454] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:06.692440  1672 raft_consensus.cc:374] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.693547  1672 raft_consensus.cc:729] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.694599  1672 consensus_queue.cc:260] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:06.691182  1674 raft_consensus.cc:348] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.698750  1674 raft_consensus.cc:374] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.699307  1674 raft_consensus.cc:729] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.703747  1672 ts_tablet_manager.cc:1398] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.091s	user 0.007s	sys 0.001s
I20240426 13:23:06.700294  1674 consensus_queue.cc:260] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.708016  1672 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:06.711014  1674 ts_tablet_manager.cc:1398] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.090s	user 0.005s	sys 0.003s
I20240426 13:23:06.715095  1298 catalog_manager.cc:5484] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:06.724381  1671 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:06.728013  1671 ts_tablet_manager.cc:1367] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.152s	user 0.006s	sys 0.006s
I20240426 13:23:06.718434  1673 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:06.728677  1673 ts_tablet_manager.cc:1367] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.211s	user 0.025s	sys 0.055s
I20240426 13:23:06.730791  1671 raft_consensus.cc:348] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.731523  1671 raft_consensus.cc:374] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.731786  1671 raft_consensus.cc:729] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.732033  1673 raft_consensus.cc:348] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.732897  1673 raft_consensus.cc:374] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.733234  1673 raft_consensus.cc:729] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.720780  2012 consensus_queue.cc:1035] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240426 13:23:06.769001  1828 raft_consensus.cc:482] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.769507  1828 raft_consensus.cc:504] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:06.775138  1828 leader_election.cc:288] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:06.780449  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.722858  1674 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:06.818022  1672 tablet_bootstrap.cc:654] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.831264  1673 consensus_queue.cc:260] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.836469  1743 consensus_queue.cc:1035] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:06.837587  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:06.837550  1673 ts_tablet_manager.cc:1398] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.109s	user 0.035s	sys 0.050s
I20240426 13:23:06.839854  1673 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
W20240426 13:23:06.846762  1367 leader_election.cc:341] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:06.847509  1366 leader_election.cc:341] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.847975  1366 leader_election.cc:302] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:06.831559  1671 consensus_queue.cc:260] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.849363  1828 raft_consensus.cc:2726] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:06.850272  1890 raft_consensus.cc:482] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.850809  1890 raft_consensus.cc:504] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.852841  1890 leader_election.cc:288] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
W20240426 13:23:06.855069  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 83935us
I20240426 13:23:06.857040  1674 tablet_bootstrap.cc:654] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.850448  1671 ts_tablet_manager.cc:1398] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.122s	user 0.005s	sys 0.000s
I20240426 13:23:06.863750  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:06.864423  1400 raft_consensus.cc:2454] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:06.865685  1441 leader_election.cc:302] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:06.866685  1919 raft_consensus.cc:2781] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.867014  1919 raft_consensus.cc:482] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.867230  1919 raft_consensus.cc:3037] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.865262  1671 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:06.874545  1919 raft_consensus.cc:504] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.875785  1671 tablet_bootstrap.cc:654] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.879037  1403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:06.879715  1403 raft_consensus.cc:3037] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.880483  2011 raft_consensus.cc:482] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.880975  2011 raft_consensus.cc:504] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.883735  2011 leader_election.cc:288] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:06.893427  2012 consensus_queue.cc:1035] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:06.884887  1743 consensus_queue.cc:1035] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240426 13:23:06.896708  1403 raft_consensus.cc:2454] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:06.898751  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:06.898793  2002 consensus_queue.cc:1035] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:23:06.901949  1441 leader_election.cc:302] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
W20240426 13:23:06.902545  1442 consensus_peers.cc:485] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:06.903578  1938 raft_consensus.cc:2781] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:06.904106  1938 raft_consensus.cc:686] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:06.904898  1938 consensus_queue.cc:237] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.906695  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:06.907150  1366 consensus_peers.cc:485] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:06.908941  1443 leader_election.cc:341] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.909174  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
W20240426 13:23:06.911819  1443 leader_election.cc:341] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:06.915493  1673 tablet_bootstrap.cc:654] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:06.922464  2012 raft_consensus.cc:482] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:06.923127  2012 raft_consensus.cc:504] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:06.924669  1672 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:06.925159  1672 ts_tablet_manager.cc:1367] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.217s	user 0.011s	sys 0.005s
I20240426 13:23:06.925592  2012 leader_election.cc:288] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:06.887996  1919 leader_election.cc:288] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:06.936107  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:06.936825  1477 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:06.938803  1680 consensus_queue.cc:1035] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:06.942109  1589 leader_election.cc:302] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:06.942904  2011 raft_consensus.cc:2781] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:06.945590  2011 raft_consensus.cc:482] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:06.946332  2011 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:06.955747  1672 raft_consensus.cc:348] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.956921  1672 raft_consensus.cc:374] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:06.957299  1672 raft_consensus.cc:729] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:06.982586  2012 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:06.958125  1672 consensus_queue.cc:260] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:06.959424  1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:06.995297  1672 ts_tablet_manager.cc:1398] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.070s	user 0.008s	sys 0.000s
I20240426 13:23:07.009357  1672 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:07.002532  1398 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:07.020073  1671 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.020529  1671 ts_tablet_manager.cc:1367] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.155s	user 0.008s	sys 0.016s
I20240426 13:23:07.040920  1671 raft_consensus.cc:348] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.041738  1671 raft_consensus.cc:374] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.042104  1671 raft_consensus.cc:729] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.045254  1624 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.019011  2011 raft_consensus.cc:504] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.047094  1671 consensus_queue.cc:260] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.052240  1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.053681  1400 raft_consensus.cc:2379] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
I20240426 13:23:07.056619  1589 leader_election.cc:302] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:07.061975  1827 consensus_queue.cc:1035] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:07.071820  2011 leader_election.cc:288] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:07.078375  1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:07.078998  1398 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.080901  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:07.081542  1476 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.090085  1673 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:07.090600  1673 ts_tablet_manager.cc:1367] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.251s	user 0.019s	sys 0.025s
I20240426 13:23:07.091939  1674 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:07.088446  1297 catalog_manager.cc:5484] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:07.092559  1674 ts_tablet_manager.cc:1367] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.370s	user 0.021s	sys 0.005s
I20240426 13:23:07.098814  1671 ts_tablet_manager.cc:1398] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.078s	user 0.009s	sys 0.000s
I20240426 13:23:07.103242  1673 raft_consensus.cc:348] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.106335  1934 raft_consensus.cc:2726] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.118497  1476 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:07.122826  1827 consensus_queue.cc:1035] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:07.103997  1673 raft_consensus.cc:374] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.131930  1673 raft_consensus.cc:729] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.132663  1673 consensus_queue.cc:260] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.134430  1673 ts_tablet_manager.cc:1398] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.000s
I20240426 13:23:07.135381  1673 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:07.130805  1994 raft_consensus.cc:482] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.136283  1994 raft_consensus.cc:504] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.135375  2002 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.137444  2002 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.138231  1994 leader_election.cc:288] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.140156  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:07.140736  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:07.140832  1549 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:07.140902  2012 consensus_queue.cc:1035] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240426 13:23:07.141968  1515 leader_election.cc:341] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.131127  1398 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:07.149801  1674 raft_consensus.cc:348] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.152410  1674 raft_consensus.cc:374] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.152724  1674 raft_consensus.cc:729] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.158464  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.148030  1367 leader_election.cc:302] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.161123  2000 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.161623  2000 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.162021  2000 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.162621  1672 tablet_bootstrap.cc:654] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.166529  1674 consensus_queue.cc:260] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.169411  1674 ts_tablet_manager.cc:1398] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.076s	user 0.006s	sys 0.000s
I20240426 13:23:07.170552  1674 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:07.151131  1671 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.175788  1674 tablet_bootstrap.cc:654] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.177232  1671 tablet_bootstrap.cc:654] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.184866  1589 leader_election.cc:302] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:07.190845  2002 leader_election.cc:288] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.192498  1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.193357  1673 tablet_bootstrap.cc:654] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.193518  2012 raft_consensus.cc:2781] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.194039  2012 raft_consensus.cc:686] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:07.194859  2012 consensus_queue.cc:237] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.200376  1934 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20240426 13:23:07.208793  1368 leader_election.cc:341] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:07.239154  1517 leader_election.cc:341] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:07.263691  1590 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 54967us
I20240426 13:23:07.260632  1517 leader_election.cc:302] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
W20240426 13:23:07.288806  1517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 51756.1us
I20240426 13:23:07.289302  1880 raft_consensus.cc:2726] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.293051  1721 raft_consensus.cc:482] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.293532  1721 raft_consensus.cc:504] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.295506  1721 leader_election.cc:288] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:07.307637  1673 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:07.308395  1673 ts_tablet_manager.cc:1367] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.173s	user 0.008s	sys 0.016s
I20240426 13:23:07.314337  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.315063  1401 raft_consensus.cc:2454] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:07.318410  2032 raft_consensus.cc:482] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.318897  2032 raft_consensus.cc:504] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.321794  1515 leader_election.cc:302] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.322955  2029 raft_consensus.cc:2781] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.323426  2029 raft_consensus.cc:482] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.323736  2029 raft_consensus.cc:3037] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.324580  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.325256  1623 raft_consensus.cc:2454] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:07.325958  2032 leader_election.cc:288] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.330230  1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:07.336247  1367 leader_election.cc:341] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.295673  2000 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.345798  1671 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.346361  1671 ts_tablet_manager.cc:1367] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.247s	user 0.017s	sys 0.005s
I20240426 13:23:07.349181  1671 raft_consensus.cc:348] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.350081  1671 raft_consensus.cc:374] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.350420  1671 raft_consensus.cc:729] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.356410  1297 catalog_manager.cc:5484] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:07.356952  1671 consensus_queue.cc:260] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.361124  1671 ts_tablet_manager.cc:1398] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20240426 13:23:07.362087  1671 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.363723  1743 consensus_queue.cc:1035] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:07.366281  1673 raft_consensus.cc:348] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.367009  1673 raft_consensus.cc:374] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.367272  1673 raft_consensus.cc:729] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.375413  2000 leader_election.cc:288] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.375763  1671 tablet_bootstrap.cc:654] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.376711  1673 consensus_queue.cc:260] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.378276  1674 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:07.379660  1674 ts_tablet_manager.cc:1367] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.209s	user 0.022s	sys 0.027s
I20240426 13:23:07.382094  1674 raft_consensus.cc:348] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.384474  1674 raft_consensus.cc:374] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.384747  1674 raft_consensus.cc:729] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.385025  2029 raft_consensus.cc:504] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.385371  1674 consensus_queue.cc:260] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.386026  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:07.386646  1548 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.387647  2029 leader_election.cc:288] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:07.389698  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:07.412592  1401 raft_consensus.cc:3037] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.427794  1548 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:07.396955  1674 ts_tablet_manager.cc:1398] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20240426 13:23:07.399587  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0863dde30e4fdab1de8f22d75fabd5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
W20240426 13:23:07.403419  1368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 85222.8us
I20240426 13:23:07.432439  1623 raft_consensus.cc:3037] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.432538  1367 leader_election.cc:302] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.433101  1674 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:07.433529  1401 raft_consensus.cc:2454] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:07.407166  1673 ts_tablet_manager.cc:1398] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.098s	user 0.006s	sys 0.001s
I20240426 13:23:07.436336  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.437718  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:07.438110  2004 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.438443  1626 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.438720  2004 raft_consensus.cc:686] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:07.439360  1623 raft_consensus.cc:2454] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:07.440722  1515 leader_election.cc:302] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.442901  1672 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:07.443562  1672 ts_tablet_manager.cc:1367] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.435s	user 0.008s	sys 0.016s
I20240426 13:23:07.442857  1880 raft_consensus.cc:2781] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.444595  1880 raft_consensus.cc:686] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:07.439640  2004 consensus_queue.cc:237] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.445669  1880 consensus_queue.cc:237] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.445696  1674 tablet_bootstrap.cc:654] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
W20240426 13:23:07.450930  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:07.446086  1672 raft_consensus.cc:348] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.451747  1672 raft_consensus.cc:374] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.452098  1672 raft_consensus.cc:729] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
W20240426 13:23:07.452746  1368 leader_election.cc:341] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.453320  1368 leader_election.cc:302] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:07.452895  1672 consensus_queue.cc:260] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.455075  2045 raft_consensus.cc:2726] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.456776  1672 ts_tablet_manager.cc:1398] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.003s
I20240426 13:23:07.463255  1672 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:07.468966  1673 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:07.469842  1672 tablet_bootstrap.cc:654] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.473465  1477 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.484169  2029 raft_consensus.cc:482] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.484719  2029 raft_consensus.cc:504] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.486809  2029 leader_election.cc:288] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:07.492587  1671 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.493114  1671 ts_tablet_manager.cc:1367] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.131s	user 0.015s	sys 0.028s
I20240426 13:23:07.495723  1671 raft_consensus.cc:348] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.496459  1671 raft_consensus.cc:374] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.496742  1671 raft_consensus.cc:729] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.497505  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:07.498200  1673 tablet_bootstrap.cc:654] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.497339  1671 consensus_queue.cc:260] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.502243  1671 ts_tablet_manager.cc:1398] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240426 13:23:07.503221  1671 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.517176  1671 tablet_bootstrap.cc:654] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.525720  1890 raft_consensus.cc:482] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.529783  1890 raft_consensus.cc:504] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.547093  1890 leader_election.cc:288] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
W20240426 13:23:07.550879  1443 consensus_peers.cc:485] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Couldn't send request to peer 948b43c3ee7f4e209c7c91300198700c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:07.555466  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:07.556094  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4efe65507a824cd88e8376761b8ebe1c" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.564988  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.573277  1399 raft_consensus.cc:2454] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
W20240426 13:23:07.574942  1442 leader_election.cc:341] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.575953  1515 leader_election.cc:302] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:07.592190  1721 raft_consensus.cc:2726] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.577963  1441 leader_election.cc:302] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:07.596735  1890 raft_consensus.cc:2781] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.597226  1890 raft_consensus.cc:482] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.580130  1672 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:07.599792  1672 ts_tablet_manager.cc:1367] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.137s	user 0.035s	sys 0.032s
I20240426 13:23:07.597599  1890 raft_consensus.cc:3037] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.609726  1672 raft_consensus.cc:348] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.610770  1672 raft_consensus.cc:374] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.611150  1672 raft_consensus.cc:729] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.580663  1674 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:07.582543  1626 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:07.583304  1671 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.620767  1671 ts_tablet_manager.cc:1367] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.118s	user 0.012s	sys 0.004s
I20240426 13:23:07.587930  1673 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:07.620049  1297 catalog_manager.cc:5484] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:07.621681  1673 ts_tablet_manager.cc:1367] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.153s	user 0.010s	sys 0.006s
I20240426 13:23:07.624459  1671 raft_consensus.cc:348] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.625227  1671 raft_consensus.cc:374] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.625531  1671 raft_consensus.cc:729] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.626576  1673 raft_consensus.cc:348] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.627277  1673 raft_consensus.cc:374] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.626924  1671 consensus_queue.cc:260] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.627528  1673 raft_consensus.cc:729] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.628192  1673 consensus_queue.cc:260] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.629158  1671 ts_tablet_manager.cc:1398] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20240426 13:23:07.629602  1673 ts_tablet_manager.cc:1398] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240426 13:23:07.630103  1671 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.630121  1674 ts_tablet_manager.cc:1367] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.197s	user 0.012s	sys 0.003s
I20240426 13:23:07.611932  1672 consensus_queue.cc:260] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.632256  1672 ts_tablet_manager.cc:1398] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20240426 13:23:07.633158  1672 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:07.632932  1674 raft_consensus.cc:348] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.633690  1674 raft_consensus.cc:374] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.633978  1674 raft_consensus.cc:729] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.634536  1674 consensus_queue.cc:260] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.637012  1674 ts_tablet_manager.cc:1398] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240426 13:23:07.637945  1674 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:07.638816  1672 tablet_bootstrap.cc:654] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.641139  1673 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:07.641619  2045 raft_consensus.cc:482] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.646107  2045 raft_consensus.cc:504] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.649015  2045 leader_election.cc:288] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:07.651299  1673 tablet_bootstrap.cc:654] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.653400  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:07.654522  1301 catalog_manager.cc:5484] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
W20240426 13:23:07.655622  1366 leader_election.cc:341] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.651233  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:07.657670  1368 leader_election.cc:341] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.658233  1368 leader_election.cc:302] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:07.660727  2004 raft_consensus.cc:2726] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.643378  1890 raft_consensus.cc:504] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.668421  1671 tablet_bootstrap.cc:654] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.674726  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
W20240426 13:23:07.676016  1442 leader_election.cc:341] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.676620  1890 leader_election.cc:288] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:07.681979  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:07.682657  1401 raft_consensus.cc:3037] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.692404  1674 tablet_bootstrap.cc:654] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.697520  1401 raft_consensus.cc:2454] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:07.702620  1441 leader_election.cc:302] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:07.703528  1890 raft_consensus.cc:2781] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.704238  1890 raft_consensus.cc:686] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:07.705260  1890 consensus_queue.cc:237] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.708461  1938 raft_consensus.cc:482] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.708962  1938 raft_consensus.cc:504] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.711431  1938 leader_election.cc:288] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:07.713423  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.714458  1401 raft_consensus.cc:2454] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:07.713447  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:07.715750  1441 leader_election.cc:302] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:07.717027  1938 raft_consensus.cc:2781] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.717235  1673 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:07.717733  1673 ts_tablet_manager.cc:1367] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.077s	user 0.012s	sys 0.003s
I20240426 13:23:07.717504  1938 raft_consensus.cc:482] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.718282  1938 raft_consensus.cc:3037] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.719288  1672 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:07.719758  1672 ts_tablet_manager.cc:1367] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.087s	user 0.033s	sys 0.024s
I20240426 13:23:07.720587  1673 raft_consensus.cc:348] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.721346  1673 raft_consensus.cc:374] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 13:23:07.721815  1442 leader_election.cc:341] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.721614  1673 raft_consensus.cc:729] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.722520  1672 raft_consensus.cc:348] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.723253  1672 raft_consensus.cc:374] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.723543  1672 raft_consensus.cc:729] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.723389  1680 consensus_queue.cc:1035] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:07.724252  1672 consensus_queue.cc:260] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.723417  1673 consensus_queue.cc:260] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.727201  1672 ts_tablet_manager.cc:1398] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20240426 13:23:07.728137  1672 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:07.733879  1671 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.734395  1671 ts_tablet_manager.cc:1367] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.104s	user 0.018s	sys 0.009s
I20240426 13:23:07.737635  1938 raft_consensus.cc:504] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.737318  1671 raft_consensus.cc:348] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.738663  1671 raft_consensus.cc:374] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.739023  1671 raft_consensus.cc:729] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.739806  1671 consensus_queue.cc:260] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.740285  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:07.740948  1399 raft_consensus.cc:3037] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.742404  1671 ts_tablet_manager.cc:1398] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20240426 13:23:07.743404  1671 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.743938  1672 tablet_bootstrap.cc:654] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.751231  1399 raft_consensus.cc:2454] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:07.753002  1938 leader_election.cc:288] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:07.755643  1674 tablet_bootstrap.cc:492] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:07.756348  1674 ts_tablet_manager.cc:1367] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.119s	user 0.014s	sys 0.001s
I20240426 13:23:07.763826  1671 tablet_bootstrap.cc:654] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.768850  1441 leader_election.cc:302] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:07.767799  1674 raft_consensus.cc:348] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.773519  1674 raft_consensus.cc:374] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.780177  1674 raft_consensus.cc:729] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.778254  1827 raft_consensus.cc:482] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.781016  1674 consensus_queue.cc:260] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.771045  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:07.773103  1938 raft_consensus.cc:2781] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.781028  1827 raft_consensus.cc:504] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
W20240426 13:23:07.787448  1442 leader_election.cc:341] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.786979  1674 ts_tablet_manager.cc:1398] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20240426 13:23:07.788381  1827 leader_election.cc:288] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.789549  1674 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:07.789888  1938 raft_consensus.cc:686] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:07.795785  1938 consensus_queue.cc:237] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.799592  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:07.804308  1477 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:07.808020  1515 leader_election.cc:302] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.808933  1721 raft_consensus.cc:2781] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.809255  1721 raft_consensus.cc:482] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.809561  1721 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.804087  1298 catalog_manager.cc:5484] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:07.818058  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.818763  1626 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:07.818800  1673 ts_tablet_manager.cc:1398] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.101s	user 0.017s	sys 0.019s
I20240426 13:23:07.820410  1673 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:07.820824  1674 tablet_bootstrap.cc:654] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.832484  1673 tablet_bootstrap.cc:654] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.834156  2029 raft_consensus.cc:482] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.834837  2029 raft_consensus.cc:504] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.853883  1674 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:07.854413  1674 ts_tablet_manager.cc:1367] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.065s	user 0.013s	sys 0.006s
I20240426 13:23:07.859637  1721 raft_consensus.cc:504] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.869912  1721 leader_election.cc:288] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:07.877477  1674 raft_consensus.cc:348] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.880920  1674 raft_consensus.cc:374] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.881335  1674 raft_consensus.cc:729] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.882809  1674 consensus_queue.cc:260] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:07.890301  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.891026  1399 raft_consensus.cc:2454] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:07.891520  1674 ts_tablet_manager.cc:1398] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.003s
I20240426 13:23:07.892716  1515 leader_election.cc:302] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.892421  1300 catalog_manager.cc:5484] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:07.884588  2029 leader_election.cc:288] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:07.872884  1973 consensus_queue.cc:1035] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240426 13:23:07.899214  1994 raft_consensus.cc:2781] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:07.904635  1672 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:07.905118  1672 ts_tablet_manager.cc:1367] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.177s	user 0.029s	sys 0.017s
I20240426 13:23:07.904292  1994 raft_consensus.cc:482] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:07.907279  1994 raft_consensus.cc:3037] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.901510  2011 raft_consensus.cc:482] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:07.908308  2011 raft_consensus.cc:504] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:07.909576  1672 raft_consensus.cc:348] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.910229  2011 leader_election.cc:288] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:07.910413  1672 raft_consensus.cc:374] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.910833  1672 raft_consensus.cc:729] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.911175  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:07.911746  1477 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.911614  1672 consensus_queue.cc:260] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.899721  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:07.916396  1674 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:07.918334  1477 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:07.919795  1515 leader_election.cc:302] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:07.922050  2029 raft_consensus.cc:2781] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:07.922565  2029 raft_consensus.cc:686] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:07.924453  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:07.924765  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:07.925726  1626 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.903869  1548 raft_consensus.cc:3037] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:07.927800  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54bf83b19a6047a7bec69cda298d5436" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:07.933014  1671 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:07.933533  1671 ts_tablet_manager.cc:1367] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.190s	user 0.023s	sys 0.007s
I20240426 13:23:07.934108  1589 leader_election.cc:302] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:07.934926  1973 raft_consensus.cc:2726] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:07.923358  2029 consensus_queue.cc:237] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
W20240426 13:23:07.937613  1517 leader_election.cc:341] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.935937  1671 raft_consensus.cc:348] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.940199  1671 raft_consensus.cc:374] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:07.943392  2000 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:07.945391  1626 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:07.940481  1671 raft_consensus.cc:729] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:07.956820  1672 ts_tablet_manager.cc:1398] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.051s	user 0.006s	sys 0.000s
I20240426 13:23:07.964757  1671 consensus_queue.cc:260] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:07.980612  1672 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:07.980841  1994 raft_consensus.cc:504] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:07.983897  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:07.986265  1671 ts_tablet_manager.cc:1398] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.000s
I20240426 13:23:07.987313  1671 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:07.987612  1973 raft_consensus.cc:482] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:23:07.989253  1367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 47684.6us
W20240426 13:23:07.989471  1517 leader_election.cc:341] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:07.988823  1994 leader_election.cc:288] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:07.997041  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:07.997615  1401 raft_consensus.cc:3037] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.003768  1672 tablet_bootstrap.cc:654] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.003893  1674 tablet_bootstrap.cc:654] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:07.988691  1973 raft_consensus.cc:504] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.006441  1973 leader_election.cc:288] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:08.012569  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:08.015604  1589 leader_election.cc:341] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.020785  2000 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.030970  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:08.031847  1551 raft_consensus.cc:2454] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:08.033808  1590 leader_election.cc:302] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:08.035548  2011 raft_consensus.cc:2781] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.040179  1673 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:08.040827  1298 catalog_manager.cc:5484] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:08.040858  1673 ts_tablet_manager.cc:1367] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.221s	user 0.016s	sys 0.006s
I20240426 13:23:08.034268  1623 raft_consensus.cc:3037] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.036562  2011 raft_consensus.cc:482] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.074945  2011 raft_consensus.cc:3037] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.084861  1673 raft_consensus.cc:348] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.085898  1673 raft_consensus.cc:374] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.086230  1673 raft_consensus.cc:729] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.087045  1673 consensus_queue.cc:260] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.072001  1671 tablet_bootstrap.cc:654] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.083619  1401 raft_consensus.cc:2454] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:08.105019  1515 leader_election.cc:302] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.106446  2011 raft_consensus.cc:504] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.106750  2029 raft_consensus.cc:2781] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.108836  2029 raft_consensus.cc:686] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:08.110105  2029 consensus_queue.cc:237] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.099273  1673 ts_tablet_manager.cc:1398] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.054s	user 0.006s	sys 0.001s
I20240426 13:23:08.117885  2070 raft_consensus.cc:482] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.119671  2070 raft_consensus.cc:504] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.121589  2070 leader_election.cc:288] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:08.124277  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:08.124816  1721 consensus_queue.cc:1035] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:23:08.126694  1367 leader_election.cc:341] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.128706  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:08.132529  1368 leader_election.cc:341] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.133028  1368 leader_election.cc:302] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:08.133858  2032 raft_consensus.cc:2726] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.138705  2066 raft_consensus.cc:482] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.139422  2066 raft_consensus.cc:504] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.174163  1673 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:08.182489  2066 leader_election.cc:288] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.183791  2011 leader_election.cc:288] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:08.189810  1674 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.195736  1674 ts_tablet_manager.cc:1367] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.292s	user 0.050s	sys 0.029s
I20240426 13:23:08.191488  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:08.209779  1551 raft_consensus.cc:3037] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.210773  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:08.192307  1721 consensus_queue.cc:1035] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.194648  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.212850  1402 raft_consensus.cc:2454] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:08.213152  1674 raft_consensus.cc:348] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.213891  1674 raft_consensus.cc:374] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.214175  1674 raft_consensus.cc:729] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
W20240426 13:23:08.214473  1589 leader_election.cc:341] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.214809  1674 consensus_queue.cc:260] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.220305  1674 ts_tablet_manager.cc:1398] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20240426 13:23:08.221292  1674 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:08.182758  1672 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:08.194797  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:08.221698  1551 raft_consensus.cc:2454] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:08.221812  1672 ts_tablet_manager.cc:1367] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.241s	user 0.012s	sys 0.008s
I20240426 13:23:08.223080  1590 leader_election.cc:302] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf
W20240426 13:23:08.223485  1517 leader_election.cc:341] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.224751  1672 raft_consensus.cc:348] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.225423  1672 raft_consensus.cc:374] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.225576  1934 raft_consensus.cc:2781] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.225592  1672 raft_consensus.cc:729] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.226198  1934 raft_consensus.cc:686] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:08.227133  1934 consensus_queue.cc:237] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.240510  1672 consensus_queue.cc:260] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.243569  1674 tablet_bootstrap.cc:654] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.242817  1672 ts_tablet_manager.cc:1398] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I20240426 13:23:08.247949  1672 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:08.262217  1672 tablet_bootstrap.cc:654] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.265888  1743 raft_consensus.cc:482] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.266424  1743 raft_consensus.cc:504] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.268710  1743 leader_election.cc:288] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.269448  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:08.270182  1402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:08.270784  1443 leader_election.cc:341] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.270856  1402 raft_consensus.cc:2454] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:08.272433  1441 leader_election.cc:302] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.274473  1743 raft_consensus.cc:2781] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.275077  1743 raft_consensus.cc:482] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.275489  1743 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:23:08.294163  1368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 85119us
I20240426 13:23:08.295876  1671 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: No bootstrap required, opened a new log
I20240426 13:23:08.296500  1671 ts_tablet_manager.cc:1367] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.309s	user 0.011s	sys 0.008s
I20240426 13:23:08.306027  2004 raft_consensus.cc:482] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.306555  2004 raft_consensus.cc:504] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.303539  1671 raft_consensus.cc:348] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.307835  1671 raft_consensus.cc:374] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.308120  1671 raft_consensus.cc:729] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.309423  2004 leader_election.cc:288] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:08.309787  1671 consensus_queue.cc:260] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.310914  1674 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.311151  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.311419  1674 ts_tablet_manager.cc:1367] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.090s	user 0.009s	sys 0.001s
I20240426 13:23:08.311966  1671 ts_tablet_manager.cc:1398] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I20240426 13:23:08.314765  1674 raft_consensus.cc:348] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.315728  1674 raft_consensus.cc:374] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 13:23:08.313053  1366 leader_election.cc:341] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.320555  1515 leader_election.cc:302] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.344328  2082 raft_consensus.cc:2781] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.344696  2082 raft_consensus.cc:482] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.345001  2082 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.330013  1673 tablet_bootstrap.cc:654] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.352818  2082 raft_consensus.cc:504] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.341274  2062 consensus_queue.cc:1035] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240426 13:23:08.331612  1743 raft_consensus.cc:504] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.319972  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:08.358829  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:08.359393  1399 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:23:08.359784  1442 consensus_peers.cc:485] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:08.360512  1442 consensus_peers.cc:485] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:08.334228  1672 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:08.364071  1672 ts_tablet_manager.cc:1367] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.116s	user 0.032s	sys 0.023s
W20240426 13:23:08.366015  1367 leader_election.cc:341] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.366430  1367 leader_election.cc:302] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:08.366576  1672 raft_consensus.cc:348] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.316006  1674 raft_consensus.cc:729] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.367306  1672 raft_consensus.cc:374] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.367645  1672 raft_consensus.cc:729] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.368464  1674 consensus_queue.cc:260] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.367501  2045 raft_consensus.cc:2726] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.355949  2082 leader_election.cc:288] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.365262  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:08.371778  1625 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.373354  1399 raft_consensus.cc:2454] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:08.376850  1672 consensus_queue.cc:260] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.377898  1515 leader_election.cc:302] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:08.378865  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:08.379196  1672 ts_tablet_manager.cc:1398] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20240426 13:23:08.379750  1673 tablet_bootstrap.cc:492] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
W20240426 13:23:08.380249  1443 leader_election.cc:341] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.380421  2066 raft_consensus.cc:2781] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.381008  2066 raft_consensus.cc:686] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:08.381747  2066 consensus_queue.cc:237] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.380402  1673 ts_tablet_manager.cc:1367] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.206s	user 0.013s	sys 0.001s
I20240426 13:23:08.383040  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:08.383471  1743 leader_election.cc:288] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.383787  1399 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.383615  1625 raft_consensus.cc:2454] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:08.373726  1674 ts_tablet_manager.cc:1398] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.062s	user 0.004s	sys 0.004s
I20240426 13:23:08.386950  1298 catalog_manager.cc:5484] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:08.388092  1673 raft_consensus.cc:348] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.388760  1673 raft_consensus.cc:374] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.389010  1673 raft_consensus.cc:729] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.391708  2083 raft_consensus.cc:482] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.392189  2083 raft_consensus.cc:504] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.394224  2083 leader_election.cc:288] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:08.395330  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:08.396848  1515 leader_election.cc:341] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.389602  1673 consensus_queue.cc:260] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.400142  1673 ts_tablet_manager.cc:1398] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20240426 13:23:08.400476  1674 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:08.401095  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
W20240426 13:23:08.403137  1517 leader_election.cc:341] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.403585  1517 leader_election.cc:302] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.406749  2082 raft_consensus.cc:2726] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.398563  1672 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:08.408732  1673 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:08.413479  1672 tablet_bootstrap.cc:654] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.415216  1673 tablet_bootstrap.cc:654] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.420936  1743 consensus_queue.cc:1035] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.423558  1399 raft_consensus.cc:2454] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:08.425290  1441 leader_election.cc:302] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.427211  1743 raft_consensus.cc:2781] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.427726  1743 raft_consensus.cc:686] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:08.428623  1743 consensus_queue.cc:237] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.441818  1297 catalog_manager.cc:5484] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:08.467273  1672 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:08.467823  1672 ts_tablet_manager.cc:1367] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.069s	user 0.026s	sys 0.014s
I20240426 13:23:08.468999  1674 tablet_bootstrap.cc:654] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.470384  1672 raft_consensus.cc:348] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.471114  1672 raft_consensus.cc:374] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.471359  1672 raft_consensus.cc:729] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.473863  1672 consensus_queue.cc:260] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.476387  1672 ts_tablet_manager.cc:1398] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20240426 13:23:08.477310  1672 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:08.479934  2083 consensus_queue.cc:1035] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:08.488646  1672 tablet_bootstrap.cc:654] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.494285  1673 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:08.494969  1673 ts_tablet_manager.cc:1367] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.086s	user 0.013s	sys 0.025s
I20240426 13:23:08.500059  1300 catalog_manager.cc:5484] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:08.492084  2083 consensus_queue.cc:1035] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240426 13:23:08.541635  1880 raft_consensus.cc:482] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.541747  1973 raft_consensus.cc:482] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.542253  1880 raft_consensus.cc:504] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.542443  1973 raft_consensus.cc:504] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.549784  1880 leader_election.cc:288] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.552459  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:08.552781  1973 leader_election.cc:288] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.555469  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
W20240426 13:23:08.558039  1517 leader_election.cc:341] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.560103  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.560779  1401 raft_consensus.cc:2454] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 0.
I20240426 13:23:08.564002  1515 leader_election.cc:302] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.568161  1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca73bd7cd06413fbad853c7ccc749f2" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.569031  1401 raft_consensus.cc:2379] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
I20240426 13:23:08.570297  1589 leader_election.cc:302] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:08.572779  1680 raft_consensus.cc:3037] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.569165  2083 raft_consensus.cc:2781] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.573942  2083 raft_consensus.cc:482] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.574966  2083 raft_consensus.cc:3037] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.581499  1674 tablet_bootstrap.cc:492] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.582020  1674 ts_tablet_manager.cc:1367] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.182s	user 0.029s	sys 0.020s
I20240426 13:23:08.587234  1680 raft_consensus.cc:2726] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.590847  1674 raft_consensus.cc:348] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.591584  1674 raft_consensus.cc:374] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.591835  1674 raft_consensus.cc:729] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.595124  1674 consensus_queue.cc:260] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.596704  1674 ts_tablet_manager.cc:1398] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20240426 13:23:08.597635  1674 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:08.598166  2083 raft_consensus.cc:504] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.600597  2083 leader_election.cc:288] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:08.601513  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:08.603243  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7af4930353e4432961278f8d6fc493e" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
W20240426 13:23:08.603773  1517 leader_election.cc:341] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:08.603866  1399 raft_consensus.cc:3037] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.606307  1673 raft_consensus.cc:348] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.607234  1673 raft_consensus.cc:374] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.607468  1673 raft_consensus.cc:729] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.608146  1673 consensus_queue.cc:260] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.613507  1674 tablet_bootstrap.cc:654] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.610762  1399 raft_consensus.cc:2454] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
W20240426 13:23:08.616094  2097 log.cc:924] Time spent T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20240426 13:23:08.616350  1515 leader_election.cc:302] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:08.617799  2083 raft_consensus.cc:2781] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.618477  2083 raft_consensus.cc:686] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:08.618683  1673 ts_tablet_manager.cc:1398] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.123s	user 0.006s	sys 0.001s
I20240426 13:23:08.619735  2083 consensus_queue.cc:237] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
W20240426 13:23:08.626910  2095 log.cc:924] Time spent T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20240426 13:23:08.638526  1297 catalog_manager.cc:5484] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:08.642997  1674 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.650589  1674 ts_tablet_manager.cc:1367] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.053s	user 0.008s	sys 0.008s
I20240426 13:23:08.653528  1674 raft_consensus.cc:348] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.654177  1674 raft_consensus.cc:374] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.654464  1674 raft_consensus.cc:729] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.655073  1674 consensus_queue.cc:260] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.655578  1672 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:08.656140  1672 ts_tablet_manager.cc:1367] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.179s	user 0.011s	sys 0.010s
I20240426 13:23:08.657264  1674 ts_tablet_manager.cc:1398] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20240426 13:23:08.658378  1674 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:08.647828  1673 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:08.648365  1880 consensus_queue.cc:1035] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.658548  1672 raft_consensus.cc:348] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.660748  1938 raft_consensus.cc:482] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.660491  2066 consensus_queue.cc:1035] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:08.661252  1938 raft_consensus.cc:504] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.660547  1672 raft_consensus.cc:374] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.663126  1672 raft_consensus.cc:729] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.663733  1672 consensus_queue.cc:260] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.663998  1938 leader_election.cc:288] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:08.667186  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:08.667549  1674 tablet_bootstrap.cc:654] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.668098  1552 raft_consensus.cc:2379] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 1: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:08.666942  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc2512b5c104b73a15c3238bfedbe80" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:08.670248  1672 ts_tablet_manager.cc:1398] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20240426 13:23:08.670117  1443 leader_election.cc:302] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:08.677176  1938 raft_consensus.cc:3037] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.685463  1674 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.686010  1674 ts_tablet_manager.cc:1367] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.005s
I20240426 13:23:08.686491  1673 tablet_bootstrap.cc:654] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.687628  1672 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: Bootstrap starting.
I20240426 13:23:08.688652  1674 raft_consensus.cc:348] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
W20240426 13:23:08.685518  1349 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:23:08.689453  1674 raft_consensus.cc:374] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.689836  1674 raft_consensus.cc:729] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.690579  1674 consensus_queue.cc:260] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.686532  1938 raft_consensus.cc:2726] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.692867  1674 ts_tablet_manager.cc:1398] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20240426 13:23:08.693859  1674 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: Bootstrap starting.
I20240426 13:23:08.694691  1673 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:08.699148  1674 tablet_bootstrap.cc:654] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.700819  1672 tablet_bootstrap.cc:654] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.711776  1673 ts_tablet_manager.cc:1367] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.064s	user 0.009s	sys 0.002s
I20240426 13:23:08.718268  2032 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.714638  1673 raft_consensus.cc:348] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.718837  2032 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.720266  1672 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: No bootstrap required, opened a new log
I20240426 13:23:08.720760  1672 ts_tablet_manager.cc:1367] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.004s
I20240426 13:23:08.722484  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:08.723341  1625 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:08.725502  1368 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:08.726601  2045 raft_consensus.cc:2781] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.727106  2045 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.727412  2032 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:08.727409  2045 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.727978  1672 raft_consensus.cc:348] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.729522  1672 raft_consensus.cc:374] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.729804  1672 raft_consensus.cc:729] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.730420  1672 consensus_queue.cc:260] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.733387  1673 raft_consensus.cc:374] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:08.733830  1673 raft_consensus.cc:729] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:08.735291  1672 ts_tablet_manager.cc:1398] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.003s
I20240426 13:23:08.743011  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.752646  2045 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.756441  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:08.757304  1625 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.744679  1673 consensus_queue.cc:260] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.747401  1550 raft_consensus.cc:3037] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.766448  1673 ts_tablet_manager.cc:1398] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.000s
I20240426 13:23:08.767751  1673 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: Bootstrap starting.
I20240426 13:23:08.768355  2082 raft_consensus.cc:482] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.769030  2082 raft_consensus.cc:504] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.771479  2082 leader_election.cc:288] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:08.773584  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.776250  1477 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 0.
I20240426 13:23:08.775470  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9766d6635658449298fdd0a3b0edae26" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.783464  1934 consensus_queue.cc:1035] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.802603  1934 consensus_queue.cc:1035] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:08.803071  1680 raft_consensus.cc:482] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.803980  1680 raft_consensus.cc:504] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.812248  1551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:08.812757  2011 raft_consensus.cc:482] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:08.813211  1551 raft_consensus.cc:2454] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:08.813571  2011 raft_consensus.cc:504] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.819005  2066 consensus_queue.cc:1035] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:23:08.820749  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.821988  1399 raft_consensus.cc:2379] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 1: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:08.824376  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.828294  1589 leader_election.cc:302] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:08.832042  1934 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.819316  2011 leader_election.cc:288] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:08.836508  1590 leader_election.cc:302] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:08.838282  2099 raft_consensus.cc:2781] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:08.838763  2099 raft_consensus.cc:482] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:08.839151  2099 raft_consensus.cc:3037] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.848227  1680 leader_election.cc:288] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:08.829082  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.849463  1477 raft_consensus.cc:2454] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 0.
I20240426 13:23:08.853881  2099 raft_consensus.cc:504] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.871080  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:08.871738  1552 raft_consensus.cc:3037] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.862640  2045 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:08.864717  1625 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:08.877813  1368 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:08.879384  1548 raft_consensus.cc:3037] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.881758  1515 leader_election.cc:302] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:08.882256  2099 leader_election.cc:288] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:08.883831  1880 raft_consensus.cc:482] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d163e7cd1714dc48ebf662957edf6cf)
I20240426 13:23:08.884366  1880 raft_consensus.cc:504] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:08.885737  2070 raft_consensus.cc:2781] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.886278  2070 raft_consensus.cc:686] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:08.887086  2070 consensus_queue.cc:237] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:08.887742  1880 leader_election.cc:288] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:08.890990  1552 raft_consensus.cc:2454] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:08.895359  1590 leader_election.cc:302] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:08.897030  2011 raft_consensus.cc:2781] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:08.898288  2011 raft_consensus.cc:686] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:08.904328  2011 consensus_queue.cc:237] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:08.908632  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:08.913770  1673 tablet_bootstrap.cc:654] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: Neither blocks nor log segments found. Creating new log.
I20240426 13:23:08.922076  1674 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: No bootstrap required, opened a new log
I20240426 13:23:08.919507  1934 raft_consensus.cc:2726] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:08.936483  1919 consensus_queue.cc:1035] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:08.937146  1674 ts_tablet_manager.cc:1367] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: Time spent bootstrapping tablet: real 0.243s	user 0.090s	sys 0.097s
I20240426 13:23:08.937590  1673 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: No bootstrap required, opened a new log
I20240426 13:23:08.918841  1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b50872bb2142fe8a7a0c2e234d24af" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:08.926784  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cbfde04d874c1bb4aac75641436065" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:08.938573  1474 raft_consensus.cc:3037] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:08.927492  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:08.938107  1673 ts_tablet_manager.cc:1367] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent bootstrapping tablet: real 0.171s	user 0.009s	sys 0.002s
I20240426 13:23:08.939756  1674 raft_consensus.cc:348] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:08.995846  1474 raft_consensus.cc:2454] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:09.003721  1674 raft_consensus.cc:374] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:09.004096  1674 raft_consensus.cc:729] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Initialized, Role: FOLLOWER
I20240426 13:23:09.004748  1674 consensus_queue.cc:260] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:09.006001  1478 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:09.006350  1674 ts_tablet_manager.cc:1398] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: Time spent starting tablet: real 0.069s	user 0.009s	sys 0.018s
W20240426 13:23:09.008123  1515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 97539.8us
I20240426 13:23:09.008512  1673 raft_consensus.cc:348] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:09.009299  1673 raft_consensus.cc:374] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:23:09.009603  1673 raft_consensus.cc:729] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Initialized, Role: FOLLOWER
I20240426 13:23:09.010221  1673 consensus_queue.cc:260] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:09.024194  1626 raft_consensus.cc:3037] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:09.026105  1919 consensus_queue.cc:1035] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.032168  1478 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 1.
I20240426 13:23:09.034397  1673 ts_tablet_manager.cc:1398] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20240426 13:23:09.037223  1297 catalog_manager.cc:5484] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:09.039294  1515 leader_election.cc:302] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:09.040620  2029 raft_consensus.cc:2726] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:09.042870  2082 raft_consensus.cc:2726] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:23:09.060808  2082 consensus_queue.cc:1035] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.065649  2029 consensus_queue.cc:1035] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.075728  1301 catalog_manager.cc:5484] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 0 to 1, leader changed from <none> to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 1 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:09.082654  1880 consensus_queue.cc:1035] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.092435  1297 catalog_manager.cc:5484] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 0 to 1, leader changed from <none> to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 1 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:09.138069  1890 raft_consensus.cc:482] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:09.138582  1890 raft_consensus.cc:504] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:09.140143  1890 leader_election.cc:288] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:09.140708  1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:09.140985  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:09.141309  1398 raft_consensus.cc:2454] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:09.141559  1548 raft_consensus.cc:2454] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 0.
I20240426 13:23:09.142325  1441 leader_election.cc:302] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:09.142933  1890 raft_consensus.cc:2781] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:23:09.143239  1890 raft_consensus.cc:482] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:09.143621  1890 raft_consensus.cc:3037] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:09.150272  1890 raft_consensus.cc:504] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:09.151751  1890 leader_election.cc:288] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:09.152500  1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:09.152598  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b081604971346229e0323b88039b905" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:09.153352  1398 raft_consensus.cc:3037] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:09.153461  1548 raft_consensus.cc:3037] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:23:09.159960  1548 raft_consensus.cc:2454] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:09.160887  1398 raft_consensus.cc:2454] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:09.161077  1442 leader_election.cc:302] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:09.161895  1890 raft_consensus.cc:2781] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:23:09.162398  1890 raft_consensus.cc:686] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:09.163154  1890 consensus_queue.cc:237] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:09.172262  1297 catalog_manager.cc:5484] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 1 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:09.293778  2070 consensus_queue.cc:1035] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.302567  2045 consensus_queue.cc:1035] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.345065  2011 consensus_queue.cc:1035] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.360735  2011 consensus_queue.cc:1035] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.593175  1938 consensus_queue.cc:1035] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:09.606977  1938 consensus_queue.cc:1035] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:23:10.623898 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
W20240426 13:23:10.626508  1515 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 473 similar messages]
W20240426 13:23:10.647987  1515 consensus_peers.cc:485] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.653551  1515 consensus_peers.cc:485] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.655683  1515 consensus_peers.cc:485] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.672889 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:23:10.673857 23796 tablet_replica.cc:330] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.674803 23796 raft_consensus.cc:2229] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.675712 23796 raft_consensus.cc:2258] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.677654 23796 tablet_replica.cc:330] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.678190 23796 raft_consensus.cc:2229] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.678529 23796 raft_consensus.cc:2258] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.680099 23796 tablet_replica.cc:330] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.680642 23796 raft_consensus.cc:2229] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.681344 23796 raft_consensus.cc:2258] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.683715 23796 tablet_replica.cc:330] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.684659 23796 raft_consensus.cc:2229] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.685017 23796 raft_consensus.cc:2258] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.687011 23796 tablet_replica.cc:330] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.687669 23796 raft_consensus.cc:2229] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.688409 23796 raft_consensus.cc:2258] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.690155 23796 tablet_replica.cc:330] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.690723 23796 raft_consensus.cc:2229] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.691201 23796 raft_consensus.cc:2258] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.691412  1441 consensus_peers.cc:485] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.693208 23796 tablet_replica.cc:330] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.694085 23796 raft_consensus.cc:2229] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.694579 23796 raft_consensus.cc:2258] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.696506 23796 tablet_replica.cc:330] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.697082 23796 raft_consensus.cc:2229] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.697474 23796 raft_consensus.cc:2258] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.699338 23796 tablet_replica.cc:330] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.700089 23796 raft_consensus.cc:2229] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.700557 23796 raft_consensus.cc:2258] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.703006 23796 tablet_replica.cc:330] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.703799 23796 raft_consensus.cc:2229] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.704203 23796 raft_consensus.cc:2258] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.706300 23796 tablet_replica.cc:330] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.707037 23796 raft_consensus.cc:2229] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.707538 23796 raft_consensus.cc:2258] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.707461  1441 consensus_peers.cc:485] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.708078  1589 consensus_peers.cc:485] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.709968 23796 tablet_replica.cc:330] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.710721 23796 raft_consensus.cc:2229] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.711127 23796 raft_consensus.cc:2258] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.713528 23796 tablet_replica.cc:330] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
W20240426 13:23:10.714152  1589 consensus_peers.cc:485] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.714399 23796 raft_consensus.cc:2229] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.715047 23796 raft_consensus.cc:2258] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.715415  1441 consensus_peers.cc:485] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.716894 23796 tablet_replica.cc:330] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.717453 23796 raft_consensus.cc:2229] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.718196 23796 raft_consensus.cc:2258] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.719800 23796 tablet_replica.cc:330] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.720340 23796 raft_consensus.cc:2229] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.720727 23796 raft_consensus.cc:2258] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.722420 23796 tablet_replica.cc:330] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.722921 23796 raft_consensus.cc:2229] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.723341 23796 raft_consensus.cc:2258] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.725462 23796 tablet_replica.cc:330] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.726132 23796 raft_consensus.cc:2229] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.727053 23796 raft_consensus.cc:2258] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.729238 23796 tablet_replica.cc:330] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.729797 23796 raft_consensus.cc:2229] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.730588 23796 raft_consensus.cc:2258] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.732331 23796 tablet_replica.cc:330] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.732882 23796 raft_consensus.cc:2229] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.733601 23796 raft_consensus.cc:2258] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.735247 23796 tablet_replica.cc:330] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.735744 23796 raft_consensus.cc:2229] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.736135 23796 raft_consensus.cc:2258] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.737759 23796 tablet_replica.cc:330] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.738334 23796 raft_consensus.cc:2229] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.738754 23796 raft_consensus.cc:2258] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.739863  1515 consensus_peers.cc:485] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.740741 23796 tablet_replica.cc:330] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.741405 23796 raft_consensus.cc:2229] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.742120 23796 raft_consensus.cc:2258] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.744470 23796 tablet_replica.cc:330] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.745036 23796 raft_consensus.cc:2229] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.745395 23796 raft_consensus.cc:2258] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.747040 23796 tablet_replica.cc:330] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.747709 23796 raft_consensus.cc:2229] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.748210 23796 raft_consensus.cc:2258] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.750334 23796 tablet_replica.cc:330] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.751071 23796 raft_consensus.cc:2229] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.751811 23796 raft_consensus.cc:2258] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.753427 23796 tablet_replica.cc:330] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.753971 23796 raft_consensus.cc:2229] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.754705 23796 raft_consensus.cc:2258] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.756475 23796 tablet_replica.cc:330] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.757071 23796 raft_consensus.cc:2229] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.757463 23796 raft_consensus.cc:2258] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.759235 23796 tablet_replica.cc:330] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.759992 23796 raft_consensus.cc:2229] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.760461 23796 raft_consensus.cc:2258] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.762727 23796 tablet_replica.cc:330] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.763536 23796 raft_consensus.cc:2229] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.764418 23796 raft_consensus.cc:2258] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.766701 23796 tablet_replica.cc:330] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.767395 23796 raft_consensus.cc:2229] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.767879 23796 raft_consensus.cc:2258] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.770073 23796 tablet_replica.cc:330] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.770774 23796 raft_consensus.cc:2229] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.771248 23796 raft_consensus.cc:2258] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.773458 23796 tablet_replica.cc:330] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.774163 23796 raft_consensus.cc:2229] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.774641 23796 raft_consensus.cc:2258] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.776667 23796 tablet_replica.cc:330] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.777160 23796 raft_consensus.cc:2229] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.777863 23796 raft_consensus.cc:2258] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.779484 23796 tablet_replica.cc:330] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.780014 23796 raft_consensus.cc:2229] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.780457 23796 raft_consensus.cc:2258] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.782446 23796 tablet_replica.cc:330] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.783004 23796 raft_consensus.cc:2229] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.783468 23796 raft_consensus.cc:2258] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.783918  1441 consensus_peers.cc:485] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.785173 23796 tablet_replica.cc:330] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.785950 23796 raft_consensus.cc:2229] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.786451 23796 raft_consensus.cc:2258] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.788554 23796 tablet_replica.cc:330] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.789110 23796 raft_consensus.cc:2229] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.789577 23796 raft_consensus.cc:2258] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.791450 23796 tablet_replica.cc:330] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.792198 23796 raft_consensus.cc:2229] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:10.793994 23796 raft_consensus.cc:2258] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.796063 23796 tablet_replica.cc:330] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.796667 23796 raft_consensus.cc:2229] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.797005 23796 raft_consensus.cc:2258] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.798662 23796 tablet_replica.cc:330] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.799113 23796 raft_consensus.cc:2229] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.799433 23796 raft_consensus.cc:2258] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.801090 23796 tablet_replica.cc:330] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.801550 23796 raft_consensus.cc:2229] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.801894 23796 raft_consensus.cc:2258] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.803685 23796 tablet_replica.cc:330] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.804157 23796 raft_consensus.cc:2229] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.804529 23796 raft_consensus.cc:2258] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.806506 23796 tablet_replica.cc:330] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.807085 23796 raft_consensus.cc:2229] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.807502 23796 raft_consensus.cc:2258] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:10.807713  1515 consensus_peers.cc:485] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.809428 23796 tablet_replica.cc:330] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.809949 23796 raft_consensus.cc:2229] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.810334 23796 raft_consensus.cc:2258] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.812212 23796 tablet_replica.cc:330] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: stopping tablet replica
I20240426 13:23:10.812613 23796 raft_consensus.cc:2229] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:10.812954 23796 raft_consensus.cc:2258] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:10.881837 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
W20240426 13:23:10.884116  1589 consensus_peers.cc:485] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.893409  1589 consensus_peers.cc:485] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.935289  1515 consensus_peers.cc:485] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.937604  1589 consensus_peers.cc:485] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.941711  1515 consensus_peers.cc:485] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.945667 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:23:10.946610  1441 consensus_peers.cc:485] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.952967  2140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:23:10.954500  2141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:23:10.957451 23796 server_base.cc:1034] running on GCE node
W20240426 13:23:10.959004  2143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:23:10.961637 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:23:10.961959 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:23:10.962165 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137790962153 us; error 0 us; skew 500 ppm
I20240426 13:23:10.962741 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240426 13:23:10.963585  1515 consensus_peers.cc:485] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.973273 23796 webserver.cc:460] Webserver started at http://127.23.61.1:46533/ using document root <none> and password file <none>
I20240426 13:23:10.974066 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:23:10.974334 23796 fs_manager.cc:362] Using existing metadata directory in first data directory
W20240426 13:23:10.978283  1515 consensus_peers.cc:485] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.979365 23796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
W20240426 13:23:10.981509  1441 consensus_peers.cc:485] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:10.986542  1589 consensus_peers.cc:485] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.988555  2148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 13:23:10.989362 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.001s
I20240426 13:23:10.989671 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
format_stamp: "Formatted at 2024-04-26 13:22:59 on dist-test-slave-jx77"
I20240426 13:23:10.990052 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240426 13:23:10.998557  1589 consensus_peers.cc:485] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:10.999783 23796 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 13:23:11.001014 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:23:11.002404 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:23:11.007035  2155 ts_tablet_manager.cc:522] Loading tablet metadata (0/45 complete)
W20240426 13:23:11.019860  1441 consensus_peers.cc:485] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.040661  1515 consensus_peers.cc:485] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.052711  1441 consensus_peers.cc:485] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.060734  1515 consensus_peers.cc:485] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.060818  1441 consensus_peers.cc:485] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.067972  1515 consensus_peers.cc:485] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.075914  1441 consensus_peers.cc:485] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.110509  1515 consensus_peers.cc:485] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.171993  1589 consensus_peers.cc:485] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:11.238072  1441 consensus_peers.cc:485] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:11.253548 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20240426 13:23:11.253911 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.248s	user 0.007s	sys 0.002s
I20240426 13:23:11.254173 23796 ts_tablet_manager.cc:574] Registering tablets (0/45 complete)
I20240426 13:23:11.258395  2155 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.274044  2155 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.274880  2155 tablet_bootstrap.cc:492] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.275481  2155 ts_tablet_manager.cc:1367] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.000s
I20240426 13:23:11.277567  2155 raft_consensus.cc:348] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.278059  2155 raft_consensus.cc:729] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.278611  2155 consensus_queue.cc:260] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.281481  2155 ts_tablet_manager.cc:1398] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.009s	sys 0.000s
I20240426 13:23:11.282119  2155 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.294322  2155 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.295192  2155 tablet_bootstrap.cc:492] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.295825  2155 ts_tablet_manager.cc:1367] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20240426 13:23:11.298027  2155 raft_consensus.cc:348] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.298677  2155 raft_consensus.cc:729] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.299253  2155 consensus_queue.cc:260] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.301044  2155 ts_tablet_manager.cc:1398] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:11.301605  2155 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.312490  2155 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.313148  2155 tablet_bootstrap.cc:492] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.313632  2155 ts_tablet_manager.cc:1367] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.004s
I20240426 13:23:11.315474  2155 raft_consensus.cc:348] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.316012  2155 raft_consensus.cc:729] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.316679  2155 consensus_queue.cc:260] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.318120  2155 ts_tablet_manager.cc:1398] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
I20240426 13:23:11.318770  2155 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.330771  2155 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.331578  2155 tablet_bootstrap.cc:492] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.332100  2155 ts_tablet_manager.cc:1367] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.013s	user 0.013s	sys 0.000s
I20240426 13:23:11.333683  2155 raft_consensus.cc:348] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.334278  2155 raft_consensus.cc:729] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.334961  2155 consensus_queue.cc:260] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.336824  2155 ts_tablet_manager.cc:1398] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:23:11.337541  2155 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.352818  2155 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.353533  2155 tablet_bootstrap.cc:492] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.354027  2155 ts_tablet_manager.cc:1367] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.004s
I20240426 13:23:11.355728  2155 raft_consensus.cc:348] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.356287  2155 raft_consensus.cc:729] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.356866  2155 consensus_queue.cc:260] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.358347  2155 ts_tablet_manager.cc:1398] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:23:11.358963  2155 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.371708  2155 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.372438  2155 tablet_bootstrap.cc:492] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.372946  2155 ts_tablet_manager.cc:1367] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.008s
I20240426 13:23:11.374598  2155 raft_consensus.cc:348] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.375097  2155 raft_consensus.cc:729] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.375619  2155 consensus_queue.cc:260] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.377166  2155 ts_tablet_manager.cc:1398] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:23:11.377825  2155 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.390609  2155 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.391541  2155 tablet_bootstrap.cc:492] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.392230  2155 ts_tablet_manager.cc:1367] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.015s	user 0.010s	sys 0.004s
I20240426 13:23:11.394294  2155 raft_consensus.cc:348] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.394989  2155 raft_consensus.cc:729] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.395598  2155 consensus_queue.cc:260] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.397607  2155 ts_tablet_manager.cc:1398] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:11.398341  2155 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.412389  2155 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.413159  2155 tablet_bootstrap.cc:492] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.413779  2155 ts_tablet_manager.cc:1367] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20240426 13:23:11.415961  2155 raft_consensus.cc:348] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.416682  2155 raft_consensus.cc:729] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.417368  2155 consensus_queue.cc:260] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.419360  2155 ts_tablet_manager.cc:1398] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20240426 13:23:11.420091  2155 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.447762  2155 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.448603  2155 tablet_bootstrap.cc:492] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.449203  2155 ts_tablet_manager.cc:1367] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.029s	user 0.014s	sys 0.004s
I20240426 13:23:11.451439  2155 raft_consensus.cc:348] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.452116  2155 raft_consensus.cc:729] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.452812  2155 consensus_queue.cc:260] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.454936  2155 ts_tablet_manager.cc:1398] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240426 13:23:11.455714  2155 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.471700 23796 ts_tablet_manager.cc:590] Registered 45 tablets
I20240426 13:23:11.472069 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.218s	user 0.205s	sys 0.005s
I20240426 13:23:11.488214  2155 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.489174  2155 tablet_bootstrap.cc:492] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.489820  2155 ts_tablet_manager.cc:1367] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.034s	user 0.022s	sys 0.008s
I20240426 13:23:11.492022  2155 raft_consensus.cc:348] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.492561  2155 raft_consensus.cc:729] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.493026  2155 consensus_queue.cc:260] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.495095  2155 ts_tablet_manager.cc:1398] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.004s
I20240426 13:23:11.495750  2155 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.515444  2155 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.516274  2155 tablet_bootstrap.cc:492] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.516739  2155 ts_tablet_manager.cc:1367] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20240426 13:23:11.518437  2155 raft_consensus.cc:348] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.519101  2155 raft_consensus.cc:729] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.519675  2155 consensus_queue.cc:260] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.521458  2155 ts_tablet_manager.cc:1398] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:11.522080  2155 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.540828  2155 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.542727  2155 tablet_bootstrap.cc:492] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.543331  2155 ts_tablet_manager.cc:1367] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.021s	user 0.019s	sys 0.000s
I20240426 13:23:11.545298  2155 raft_consensus.cc:348] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.545954  2155 raft_consensus.cc:729] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.546542  2155 consensus_queue.cc:260] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.553637  2155 ts_tablet_manager.cc:1398] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20240426 13:23:11.554386  2155 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.628309  2156 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.629150  2156 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.634253  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:11.639092  1515 leader_election.cc:334] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111)
I20240426 13:23:11.639685  1515 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:11.641324  2083 raft_consensus.cc:2726] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:11.631769  2156 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:11.657860  2155 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.659137  2155 tablet_bootstrap.cc:492] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.660022  2155 ts_tablet_manager.cc:1367] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.106s	user 0.029s	sys 0.032s
I20240426 13:23:11.662994  2155 raft_consensus.cc:348] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:11.663875  2155 raft_consensus.cc:729] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.670426  2155 consensus_queue.cc:260] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:11.673522  2155 ts_tablet_manager.cc:1398] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20240426 13:23:11.674397  2155 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.692497  2155 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.693331  2155 tablet_bootstrap.cc:492] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.693857  2155 ts_tablet_manager.cc:1367] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.003s
I20240426 13:23:11.695525  2155 raft_consensus.cc:348] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.696199  2155 raft_consensus.cc:729] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.696899  2155 consensus_queue.cc:260] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.698900  2155 ts_tablet_manager.cc:1398] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:11.699641  2155 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.721421  2155 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.722481  2155 tablet_bootstrap.cc:492] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.723179  2155 ts_tablet_manager.cc:1367] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.004s
I20240426 13:23:11.725875  2155 raft_consensus.cc:348] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.726779  2155 raft_consensus.cc:729] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.727694  2155 consensus_queue.cc:260] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.729779  2155 ts_tablet_manager.cc:1398] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:23:11.730499  2155 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.743597  2155 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.744597  2155 tablet_bootstrap.cc:492] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.745285  2155 ts_tablet_manager.cc:1367] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.003s
I20240426 13:23:11.747614  2155 raft_consensus.cc:348] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.748404  2155 raft_consensus.cc:729] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.749116  2155 consensus_queue.cc:260] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.751441  2155 ts_tablet_manager.cc:1398] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240426 13:23:11.752244  2155 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.767058  2155 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.767946  2155 tablet_bootstrap.cc:492] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.768575 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:43845
I20240426 13:23:11.768631  2155 ts_tablet_manager.cc:1367] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.000s
I20240426 13:23:11.768723  2233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:43845 every 8 connection(s)
I20240426 13:23:11.771642  2155 raft_consensus.cc:348] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.772377  2155 raft_consensus.cc:729] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.773175  2155 consensus_queue.cc:260] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.774724  2155 ts_tablet_manager.cc:1398] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:23:11.775504  2155 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.789404  2155 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.790328  2155 tablet_bootstrap.cc:492] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.791013  2155 ts_tablet_manager.cc:1367] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.003s
I20240426 13:23:11.793645  2155 raft_consensus.cc:348] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.794420  2155 raft_consensus.cc:729] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.794982  2155 consensus_queue.cc:260] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.796978  2155 ts_tablet_manager.cc:1398] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240426 13:23:11.797839  2155 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.825583  1938 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.826920  2155 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.828395  2155 tablet_bootstrap.cc:492] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.829299  2155 ts_tablet_manager.cc:1367] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.016s
I20240426 13:23:11.832526  2155 raft_consensus.cc:348] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.833474  2155 raft_consensus.cc:729] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.834391  2155 consensus_queue.cc:260] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.836647  2155 ts_tablet_manager.cc:1398] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240426 13:23:11.837839  2155 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.826388  1938 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.838790  2011 raft_consensus.cc:482] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.839395  2011 raft_consensus.cc:504] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.841372  2011 leader_election.cc:288] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.842711  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:11.852136  2062 raft_consensus.cc:482] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.852795  2062 raft_consensus.cc:504] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.853542  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:11.853716  2170 raft_consensus.cc:482] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.854528  2170 raft_consensus.cc:504] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.855453  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:11.857448  2170 leader_election.cc:288] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.858518  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:11.858803  2062 leader_election.cc:288] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.865556  2155 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.866685  2155 tablet_bootstrap.cc:492] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.867475  2155 ts_tablet_manager.cc:1367] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.000s
I20240426 13:23:11.872078  2156 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.872800  2156 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.875319  2156 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:11.878576  2155 raft_consensus.cc:348] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.879333  2155 raft_consensus.cc:729] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.880283  1938 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:11.877126  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:11.889621  2155 consensus_queue.cc:260] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.892305  2062 raft_consensus.cc:482] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.892654  2155 ts_tablet_manager.cc:1398] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.001s
I20240426 13:23:11.892971  2062 raft_consensus.cc:504] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.893884  2155 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.897984  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:11.899642  2157 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.901032  2157 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.905071  2062 leader_election.cc:288] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:11.907403  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:11.903717  2157 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.919138  2155 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.920486  2155 tablet_bootstrap.cc:492] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.921350  2155 ts_tablet_manager.cc:1367] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.008s
I20240426 13:23:11.924575  2155 raft_consensus.cc:348] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.925552  2155 raft_consensus.cc:729] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.926190  2155 consensus_queue.cc:260] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.928619  2155 ts_tablet_manager.cc:1398] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240426 13:23:11.929479  2155 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.948244  1938 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.949064  1938 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.953076  2155 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.954362  2155 tablet_bootstrap.cc:492] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.955215  2155 ts_tablet_manager.cc:1367] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.026s	user 0.022s	sys 0.000s
I20240426 13:23:11.958052  2155 raft_consensus.cc:348] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.958765  2155 raft_consensus.cc:729] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:11.959390  2155 consensus_queue.cc:260] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.959893  2158 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.960484  2158 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.961436  2155 ts_tablet_manager.cc:1398] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20240426 13:23:11.962322  2155 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:11.962543  2158 leader_election.cc:288] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.966826  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:11.973109  2170 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.973594  2170 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.975343  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:11.976218  2170 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:11.978652  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:11.980072  1548 raft_consensus.cc:2454] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:11.981004  2156 raft_consensus.cc:482] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.981550  1442 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:11.982693  1880 raft_consensus.cc:482] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.983315  1880 raft_consensus.cc:504] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:11.985594  1880 leader_election.cc:288] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:11.988420  2244 raft_consensus.cc:2781] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:11.988822  2244 raft_consensus.cc:482] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.989148  2244 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:11.981969  2156 raft_consensus.cc:504] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.992272  2011 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:11.995241  2155 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:11.998832  2155 tablet_bootstrap.cc:492] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:11.999686  2155 ts_tablet_manager.cc:1367] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.005s
I20240426 13:23:12.001096  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:12.001718  1625 raft_consensus.cc:2454] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.002053  2155 raft_consensus.cc:348] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.002768  2243 raft_consensus.cc:482] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.002781  2155 raft_consensus.cc:729] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:12.003485  2243 raft_consensus.cc:504] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.003911  2155 consensus_queue.cc:260] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.004906  1517 leader_election.cc:302] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.005905  2155 ts_tablet_manager.cc:1398] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240426 13:23:12.006712  2155 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:12.006821  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.007217  1880 raft_consensus.cc:2781] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.007457  1474 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.007827  1880 raft_consensus.cc:482] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.008600  1880 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.008801  1589 leader_election.cc:302] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.009665  2243 leader_election.cc:288] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.009716  2158 raft_consensus.cc:2781] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.010365  2158 raft_consensus.cc:482] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.010953  2158 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.014174  2170 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.013909  2244 raft_consensus.cc:504] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.015290  2170 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:11.992905  2011 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.020045  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:12.020388  2244 leader_election.cc:288] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.020823  1548 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.022821  2011 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.023953  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.024645  1474 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.025797  1589 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.026777  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.027547  2155 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:12.027432  1474 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.028798  2155 tablet_bootstrap.cc:492] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:12.029474  2155 ts_tablet_manager.cc:1367] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.023s	user 0.020s	sys 0.001s
I20240426 13:23:12.029299  1515 leader_election.cc:302] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.030576  2083 raft_consensus.cc:2781] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.031066  2083 raft_consensus.cc:482] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.031533  2083 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.032487  2155 raft_consensus.cc:348] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.033505  2155 raft_consensus.cc:729] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:12.034328  2155 consensus_queue.cc:260] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.037099  2155 ts_tablet_manager.cc:1398] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240426 13:23:12.038159  2155 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:12.041977  2243 raft_consensus.cc:2781] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.042464  2243 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.042836  2243 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.055112  2243 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.058349  2243 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.058281  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.061168  1474 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:23:12.056129  1443 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 83396.6us
I20240426 13:23:12.058279  2155 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:12.068831  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:12.069340  2155 tablet_bootstrap.cc:492] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:12.069975  1625 raft_consensus.cc:2379] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:12.070293  2155 ts_tablet_manager.cc:1367] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.003s
I20240426 13:23:12.071563  2157 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.072117  2157 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.072952  2155 raft_consensus.cc:348] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.073664  2155 raft_consensus.cc:729] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:12.074295  2155 consensus_queue.cc:260] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.074749  2157 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.076872  1474 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.078267  1589 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.078977  2155 ts_tablet_manager.cc:1398] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20240426 13:23:12.079331  2243 raft_consensus.cc:2781] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.079800  2155 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:12.079811  2243 raft_consensus.cc:686] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.080900  2243 consensus_queue.cc:237] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.085501  2156 leader_election.cc:288] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.086365  2156 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.087002  2156 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:11.973567  1938 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.122656  2243 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.123391  2243 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.126097  2243 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.127317  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.128284  1474 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.129776  1589 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.130864  2243 raft_consensus.cc:2781] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.131358  2243 raft_consensus.cc:482] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.131825  2243 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.131830  2170 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.142482  2243 raft_consensus.cc:504] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.147076  2139 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.147785  2139 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.148633  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.149231  1474 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.150401  2139 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.152602  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.153237  1477 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.154243  1589 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.154660  2157 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.155351  2157 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.158440  2157 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.158478  1474 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.160241  1589 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.160853  2139 raft_consensus.cc:2781] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.161737  2139 raft_consensus.cc:482] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.161866  2243 leader_election.cc:288] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.162468  2139 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.162734  2243 raft_consensus.cc:2781] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.164808  2243 raft_consensus.cc:686] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.165949  2243 consensus_queue.cc:237] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.047696  2083 raft_consensus.cc:504] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.176564  2083 leader_election.cc:288] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.187709  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:12.189236  2011 raft_consensus.cc:482] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.190049  2011 raft_consensus.cc:504] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.192965  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:12.193863  1550 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:12.194136  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.195036  1474 raft_consensus.cc:2454] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.196415  1589 leader_election.cc:302] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.197343  2011 leader_election.cc:288] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.197345  2254 raft_consensus.cc:2781] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.197948  2254 raft_consensus.cc:482] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.198421  2254 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.199983  2251 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.207527  2254 raft_consensus.cc:504] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.210958  2170 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.212234  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.212754  2234 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34177
I20240426 13:23:12.217113  1474 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.217307  2234 heartbeater.cc:464] Registering TS with master...
I20240426 13:23:12.219790  2234 heartbeater.cc:511] Master 127.23.61.62:34177 requested a full tablet report, sending...
I20240426 13:23:12.224251  1474 raft_consensus.cc:2454] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.225020  2254 leader_election.cc:288] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.227124  2170 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.233294  1298 catalog_manager.cc:5484] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:12.200584  2251 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.243975  1474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
W20240426 13:23:12.263435  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 44503.7us
I20240426 13:23:12.244529  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.268723  1548 raft_consensus.cc:2454] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:12.330309  1477 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.320015  1474 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.328050  1880 raft_consensus.cc:504] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.240815  1442 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.338145  1880 leader_election.cc:288] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.339423  1938 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:23:12.339949  1442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 98346.5us
I20240426 13:23:12.340148  1938 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.340648  1938 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.240733  2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.244122  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.280231  2262 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.345211  2199 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:12.263823  2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.345500  2262 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.347962  2262 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.351611  1477 raft_consensus.cc:2454] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:12.353894  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea" is_pre_election: true
I20240426 13:23:12.354629  1550 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:12.354759  1515 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.356451  2083 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.237150  2139 raft_consensus.cc:504] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.255671  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.255519  2248 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.357260  2083 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.252933  2170 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.248059  2156 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.360746  1589 leader_election.cc:302] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.241366  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.368243  1625 raft_consensus.cc:2379] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:12.249207  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.365494  1938 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.362462  2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.267925  2205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.243561  2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.370121  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.368898  2262 raft_consensus.cc:2781] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.366357  2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.385680  2262 raft_consensus.cc:686] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.386876  2262 consensus_queue.cc:237] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.389518  2139 leader_election.cc:288] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.375172  2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.395064  2206 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
W20240426 13:23:12.376235  1441 leader_election.cc:341] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.395939  1441 leader_election.cc:302] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.397943  1441 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.377061  2251 leader_election.cc:288] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.377781  2198 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:12.378630  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc8321cda547cc94f2dcdd71507e28" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.379388  1938 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.381013  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.366909  2248 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.381448  1442 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.381552  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:12.240320  2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.416972  2204 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 1.
I20240426 13:23:12.417325  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.417888  2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.418361  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191d125f2e474efabc176055155755b0" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.276330  2158 raft_consensus.cc:504] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.383899  1477 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.419363  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442e14b352624bdbb04e9a3fdc445d32" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.419631  2062 raft_consensus.cc:2781] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.420487  2158 leader_election.cc:288] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.420850  2062 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.421283  2062 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.422974  2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.423643  2203 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.393530  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c" is_pre_election: true
I20240426 13:23:12.424314  1626 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.424401  1623 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.425357  1589 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.409863  1938 raft_consensus.cc:2781] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.425988  1938 raft_consensus.cc:686] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:12.412843  1442 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.426724  1517 leader_election.cc:302] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.405779  2170 raft_consensus.cc:2726] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.415802  1473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.428387  2156 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.429183  1473 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.429484  2156 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.430007  2156 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.431156  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.431730  1475 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.432173  2244 raft_consensus.cc:2781] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.432401  1301 ts_manager.cc:194] Re-registered known tserver with Master: 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.432809  2244 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.433353  2244 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.432123  2259 raft_consensus.cc:2781] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.435675  2259 raft_consensus.cc:482] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.436012  2259 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.418491  2204 raft_consensus.cc:2454] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
W20240426 13:23:12.409613  1441 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.405483  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.453843  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.364814  2083 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.455533  2062 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.451082  2248 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.451500  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.426730  1938 consensus_queue.cc:237] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.457348  1589 leader_election.cc:302] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: b156dff0c7b441c6b6f601006d0fcaea
W20240426 13:23:12.467139  1589 leader_election.cc:341] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.468183  2243 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.468546  2243 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.468899  2243 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.469838  1623 raft_consensus.cc:2454] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:12.464874  2155 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:12.473029  2155 tablet_bootstrap.cc:492] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:12.473697  2155 ts_tablet_manager.cc:1367] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.394s	user 0.066s	sys 0.118s
I20240426 13:23:12.475190  2243 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.454952  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b84898ee62e4f8eb0fb983c9c927fe0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
W20240426 13:23:12.476549  1589 leader_election.cc:341] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.476935  2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.477838  1548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:12.478633  1517 leader_election.cc:302] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.479403  2248 raft_consensus.cc:2781] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election won for term 2
W20240426 13:23:12.457849  1441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 58722.4us
I20240426 13:23:12.479880  2248 raft_consensus.cc:686] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
W20240426 13:23:12.481571  1441 leader_election.cc:341] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.481743  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eec2b6b55d4310bfd18eb2f7b093a5" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.481982  2248 consensus_queue.cc:237] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.480455  2243 leader_election.cc:288] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.486575  2083 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
W20240426 13:23:12.488411  1589 leader_election.cc:341] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.476017  2155 raft_consensus.cc:348] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.482013  1441 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
W20240426 13:23:12.492053  1589 leader_election.cc:341] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.492596  1441 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.489784  2155 raft_consensus.cc:729] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
W20240426 13:23:12.494457  1442 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 67228us
I20240426 13:23:12.493948  2155 consensus_queue.cc:260] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.495832  2083 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.498557  1515 leader_election.cc:302] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.499284  1880 raft_consensus.cc:2781] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.499840  1880 raft_consensus.cc:686] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:12.500813  1880 consensus_queue.cc:237] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.485580  2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
W20240426 13:23:12.507719  1441 leader_election.cc:341] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.508358  1441 leader_election.cc:302] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.517488  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:12.514117  2155 ts_tablet_manager.cc:1398] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.001s
I20240426 13:23:12.495246  2062 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
W20240426 13:23:12.525187  1589 leader_election.cc:341] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.525730  1473 raft_consensus.cc:2454] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.526571  2170 raft_consensus.cc:2726] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.527731  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.528972  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.529858  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.530472  2202 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.505223  2198 raft_consensus.cc:2454] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.509133  2244 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.533725  2244 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.533453  2259 raft_consensus.cc:504] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.505985  1475 raft_consensus.cc:2454] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.522728  2197 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 1.
I20240426 13:23:12.531378  2156 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.535665  2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.523823  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:12.543254  2197 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.543144  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.544540  1550 raft_consensus.cc:2379] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
I20240426 13:23:12.545578  1548 raft_consensus.cc:2379] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 2: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
I20240426 13:23:12.548384  1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:12.556134  1623 raft_consensus.cc:2379] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 2: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:12.556502  2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f73751c65ca4d568467edaf86391737" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.548719  2156 leader_election.cc:288] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:12.549307  2198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.560724  2198 raft_consensus.cc:3037] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.535451  2259 leader_election.cc:288] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.551119  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.551232  2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.551720  1626 raft_consensus.cc:2379] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
W20240426 13:23:12.552270  1515 leader_election.cc:341] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.564235  1515 leader_election.cc:302] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:12.559509  2155 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
W20240426 13:23:12.562995  1589 leader_election.cc:341] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.565382  1515 leader_election.cc:341] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.565657  1589 leader_election.cc:302] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
W20240426 13:23:12.566126  1515 leader_election.cc:341] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.566763  1515 leader_election.cc:341] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.567234  1515 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.568753  1515 leader_election.cc:341] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.571550  1441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 62856.7us
W20240426 13:23:12.572362  1441 leader_election.cc:341] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.572888  1441 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.575235  2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1fd3ed641c443095bc506c4c0107e7" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.576454  2268 raft_consensus.cc:2726] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:23:12.577534  1441 leader_election.cc:341] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:23:12.577632  1589 leader_election.cc:341] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.577955  1441 leader_election.cc:302] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:12.579492  2268 raft_consensus.cc:2726] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:23:12.579865  1441 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.582129  2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:12.583530  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:12.584194  1478 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.585997  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:12.586629  1550 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:23:12.590927  1589 leader_election.cc:341] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.591822  1441 leader_election.cc:302] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.593263  1515 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.594810  2251 raft_consensus.cc:2781] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.595165  2251 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.562652  2199 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:12.595530  2251 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.598564  2197 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.598837  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.599787  1589 leader_election.cc:302] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:12.583647  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.600796  1475 raft_consensus.cc:2379] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 2: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:12.602907  2198 raft_consensus.cc:2454] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 2.
I20240426 13:23:12.603327  2170 raft_consensus.cc:2741] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 13:23:12.569983  2259 raft_consensus.cc:2726] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.570659  2244 raft_consensus.cc:2726] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.605031  2243 raft_consensus.cc:2781] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.605662  2243 raft_consensus.cc:686] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.606484  2243 consensus_queue.cc:237] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
W20240426 13:23:12.605080  1515 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.609661  1515 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:12.610858  2250 raft_consensus.cc:2726] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.613426  1475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.614595  1475 raft_consensus.cc:2379] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 948b43c3ee7f4e209c7c91300198700c in current term 2: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:12.615620  1515 leader_election.cc:302] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b156dff0c7b441c6b6f601006d0fcaea; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.619920  1441 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:12.621263  1589 leader_election.cc:302] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:12.623521  2262 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.624065  2262 raft_consensus.cc:686] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.624895  2262 consensus_queue.cc:237] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.625592  1550 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:12.623024  1589 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:12.628995  1589 leader_election.cc:302] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:12.629890  1442 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:12.631237  1938 raft_consensus.cc:2781] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.631680  1938 raft_consensus.cc:686] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:12.630302  2083 raft_consensus.cc:2726] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.632834  1938 consensus_queue.cc:237] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.570180  2156 raft_consensus.cc:2726] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.631903  2139 raft_consensus.cc:2781] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.637568  1478 raft_consensus.cc:1261] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:12.646225  2157 raft_consensus.cc:2781] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:23:12.646659  2157 raft_consensus.cc:482] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 56e4ab2f7d204691a8f5a355a66a76a4)
I20240426 13:23:12.647050  2157 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.650624  1297 catalog_manager.cc:5484] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 2 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
W20240426 13:23:12.652882  1441 leader_election.cc:341] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:23:12.654053  1441 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:12.644183  2260 raft_consensus.cc:2781] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:12.661828  2244 raft_consensus.cc:2726] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.664201  2139 raft_consensus.cc:686] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.689867  2260 raft_consensus.cc:686] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:12.709887  2155 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:12.713142  2155 tablet_bootstrap.cc:492] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:12.714391  2155 ts_tablet_manager.cc:1367] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.155s	user 0.014s	sys 0.029s
I20240426 13:23:12.689303  2259 consensus_queue.cc:1035] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:23:12.721298  2155 raft_consensus.cc:348] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.722092  2155 raft_consensus.cc:729] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:12.718456  2139 consensus_queue.cc:237] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.722720  2155 consensus_queue.cc:260] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.727603  2155 ts_tablet_manager.cc:1398] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20240426 13:23:12.726099  2251 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.736644  2251 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:12.721318  2260 consensus_queue.cc:237] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:12.741951  2155 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:12.720532  2157 raft_consensus.cc:504] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:12.747591  2157 leader_election.cc:288] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:12.752449  2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea2d370e87e4c90ad5c68256ea2b940" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.753226  2206 raft_consensus.cc:2379] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate 948b43c3ee7f4e209c7c91300198700c in this term.
I20240426 13:23:12.753123  2200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.753875  2200 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:23:12.808452  1590 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 89338.4us
I20240426 13:23:12.825577  1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:12.830629  1478 raft_consensus.cc:2379] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b156dff0c7b441c6b6f601006d0fcaea in current term 2: Already voted for candidate 0d163e7cd1714dc48ebf662957edf6cf in this term.
I20240426 13:23:12.854483  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:12.855685  1552 raft_consensus.cc:2379] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d163e7cd1714dc48ebf662957edf6cf in current term 2: Already voted for candidate b156dff0c7b441c6b6f601006d0fcaea in this term.
W20240426 13:23:12.872646  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 183520us
W20240426 13:23:12.886071  1443 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:34539, user_credentials={real_user=slave}} blocked reactor thread for 179777us
I20240426 13:23:12.887135  2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.887380  1443 leader_election.cc:302] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:12.888116  2199 raft_consensus.cc:3037] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.889045  2244 raft_consensus.cc:2726] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:23:12.903318  2155 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:12.905026  2155 tablet_bootstrap.cc:492] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:12.906306  2155 ts_tablet_manager.cc:1367] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.165s	user 0.014s	sys 0.005s
I20240426 13:23:12.919296  1298 catalog_manager.cc:5484] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:12.924991  1297 catalog_manager.cc:5484] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:12.913367  2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919c4656a25c42e6a02d8d0a8d1f7f59" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:12.920228  1300 catalog_manager.cc:5484] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 2 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:12.921077  2205 raft_consensus.cc:3037] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:12.927646  1297 catalog_manager.cc:5484] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:12.945200  2155 raft_consensus.cc:348] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.946508  1475 raft_consensus.cc:1261] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:12.940382  1297 catalog_manager.cc:5484] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:12.983577  2155 raft_consensus.cc:729] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:12.953353  1441 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf; no voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:12.998338  2155 consensus_queue.cc:260] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:12.976454  1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38184
I20240426 13:23:12.999394  2244 raft_consensus.cc:2726] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 13:23:12.967319  1589 consensus_peers.cc:485] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:12.983080  1515 consensus_peers.cc:485] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:13.001031  1515 consensus_peers.cc:485] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20240426 13:23:13.002341  1515 consensus_peers.cc:485] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:13.011935  2200 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:13.013147  1622 raft_consensus.cc:1261] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:12.979761  2199 raft_consensus.cc:2454] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:12.949658  1474 raft_consensus.cc:1261] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:23:12.998783  1441 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 45729.4us
I20240426 13:23:13.015604  2155 ts_tablet_manager.cc:1398] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.091s	user 0.005s	sys 0.003s
I20240426 13:23:13.017351  2155 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:12.984266  2205 raft_consensus.cc:1261] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.026899  2198 raft_consensus.cc:3037] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:13.038300  2234 heartbeater.cc:503] Master 127.23.61.62:34177 was elected leader, sending a full tablet report...
I20240426 13:23:13.039026  1515 leader_election.cc:302] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:13.041702  2156 raft_consensus.cc:2781] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:13.042279  2156 raft_consensus.cc:686] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:13.048717  2156 consensus_queue.cc:237] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.053261  1550 raft_consensus.cc:1261] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.056733  2244 consensus_queue.cc:1035] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20240426 13:23:13.061856  1441 consensus_peers.cc:485] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:13.056931  1297 catalog_manager.cc:5484] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:13.079062  2083 consensus_queue.cc:1035] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.073190  1297 catalog_manager.cc:5484] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:13.085214  2254 consensus_queue.cc:1035] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.085561  2011 consensus_queue.cc:1035] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.087002  2155 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.088086  2155 tablet_bootstrap.cc:492] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.088853  2155 ts_tablet_manager.cc:1367] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.072s	user 0.015s	sys 0.010s
I20240426 13:23:13.096695  1297 catalog_manager.cc:5484] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 2 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:13.099252  2155 raft_consensus.cc:348] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.072764  1298 catalog_manager.cc:5484] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:13.100196  2155 raft_consensus.cc:729] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.086423  2198 raft_consensus.cc:1261] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.103116  2155 consensus_queue.cc:260] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.108745  2155 ts_tablet_manager.cc:1398] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.005s
I20240426 13:23:13.116503  2155 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.131647  2205 raft_consensus.cc:1261] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.133309  1300 catalog_manager.cc:5484] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 2, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 2 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:13.134464  2277 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.161960  1473 raft_consensus.cc:1261] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.163038  2286 consensus_queue.cc:1035] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.175402  2156 consensus_queue.cc:1035] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:23:13.201208  2286 consensus_queue.cc:1035] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240426 13:23:13.195333  1547 raft_consensus.cc:1261] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:23:13.217337  1515 consensus_peers.cc:485] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20240426 13:23:13.211400  1589 consensus_peers.cc:485] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:13.232875  2062 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.263808  2198 raft_consensus.cc:1261] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.265000  1547 raft_consensus.cc:1261] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.266757  2011 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:13.267819  1474 raft_consensus.cc:1261] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.300209  2011 consensus_queue.cc:1035] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:13.327428  1475 raft_consensus.cc:1261] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.328459  1474 raft_consensus.cc:1261] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:23:13.324707  1589 consensus_peers.cc:485] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240426 13:23:13.369675  2155 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240426 13:23:13.390780  1589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:43845, user_credentials={real_user=slave}} blocked reactor thread for 47487.6us
I20240426 13:23:13.376966  2155 tablet_bootstrap.cc:492] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.392307  2155 ts_tablet_manager.cc:1367] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.276s	user 0.014s	sys 0.009s
W20240426 13:23:13.392364  1515 consensus_peers.cc:485] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240426 13:23:13.397775  2158 consensus_queue.cc:1035] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240426 13:23:13.398048  2198 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:13.403471  2291 consensus_queue.cc:1035] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240426 13:23:13.404459  1589 consensus_peers.cc:485] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:13.416908  2155 raft_consensus.cc:348] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.417901  2155 raft_consensus.cc:729] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.418751  2155 consensus_queue.cc:260] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.423203  2155 ts_tablet_manager.cc:1398] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.030s	user 0.004s	sys 0.001s
I20240426 13:23:13.425594  2254 consensus_queue.cc:1035] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.449549  2155 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.501215  2201 raft_consensus.cc:1261] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.504833  2260 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.561007  2203 raft_consensus.cc:3037] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:13.569633  1476 raft_consensus.cc:1261] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.572108  2251 consensus_queue.cc:1035] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:13.614600  2201 raft_consensus.cc:1261] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:13.617902  2247 consensus_queue.cc:1035] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240426 13:23:13.642521  1589 consensus_peers.cc:485] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20240426 13:23:13.663017  1515 consensus_peers.cc:485] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240426 13:23:13.681354  2199 raft_consensus.cc:3037] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:23:13.693116  1441 consensus_peers.cc:485] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240426 13:23:13.697273  2155 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.700839  2155 tablet_bootstrap.cc:492] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.701597  2155 ts_tablet_manager.cc:1367] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.273s	user 0.019s	sys 0.000s
I20240426 13:23:13.705171  2155 raft_consensus.cc:348] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:13.706142  2155 raft_consensus.cc:729] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.707191  2155 consensus_queue.cc:260] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:13.710767  2155 ts_tablet_manager.cc:1398] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20240426 13:23:13.711818  2155 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.745433  2155 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.746150  2155 tablet_bootstrap.cc:492] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.746874  2155 ts_tablet_manager.cc:1367] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.035s	user 0.008s	sys 0.016s
I20240426 13:23:13.749240  2155 raft_consensus.cc:348] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:13.749970  2155 raft_consensus.cc:729] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.750586  2155 consensus_queue.cc:260] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:13.753085  2155 ts_tablet_manager.cc:1398] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20240426 13:23:13.753753  2155 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.772388  2155 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.773272  2155 tablet_bootstrap.cc:492] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.773887  2155 ts_tablet_manager.cc:1367] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.004s
I20240426 13:23:13.776149  2155 raft_consensus.cc:348] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.776790  2155 raft_consensus.cc:729] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.777390  2155 consensus_queue.cc:260] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.779446  2155 ts_tablet_manager.cc:1398] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:13.780201  2155 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.802699  2155 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.803700  2155 tablet_bootstrap.cc:492] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.804409  2155 ts_tablet_manager.cc:1367] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.003s
I20240426 13:23:13.807029  2155 raft_consensus.cc:348] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.807754  2155 raft_consensus.cc:729] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.808380  2155 consensus_queue.cc:260] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.810436  2155 ts_tablet_manager.cc:1398] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20240426 13:23:13.811268  2155 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.835412  2155 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.836405  2155 tablet_bootstrap.cc:492] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.837065  2155 ts_tablet_manager.cc:1367] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.011s
I20240426 13:23:13.839573  2155 raft_consensus.cc:348] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.840380  2155 raft_consensus.cc:729] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.841235  2155 consensus_queue.cc:260] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.843744  2155 ts_tablet_manager.cc:1398] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240426 13:23:13.844595  2155 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.883765  2155 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.885380  2155 tablet_bootstrap.cc:492] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.886094  2155 ts_tablet_manager.cc:1367] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.020s
I20240426 13:23:13.888589  2155 raft_consensus.cc:348] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.889322  2155 raft_consensus.cc:729] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.890051  2155 consensus_queue.cc:260] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.892392  2155 ts_tablet_manager.cc:1398] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20240426 13:23:13.893208  2155 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
W20240426 13:23:13.900352  1515 consensus_peers.cc:485] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240426 13:23:13.922536  2155 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.923624  2155 tablet_bootstrap.cc:492] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.924373  2155 ts_tablet_manager.cc:1367] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.016s
I20240426 13:23:13.926707  2155 raft_consensus.cc:348] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.927368  2155 raft_consensus.cc:729] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.928050  2155 consensus_queue.cc:260] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:13.935055  2155 ts_tablet_manager.cc:1398] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20240426 13:23:13.935853  2155 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.958673  2155 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.959556  2155 tablet_bootstrap.cc:492] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:13.960278  2155 ts_tablet_manager.cc:1367] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.006s
I20240426 13:23:13.962618  2155 raft_consensus.cc:348] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.963078  2155 raft_consensus.cc:729] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:13.963600  2155 consensus_queue.cc:260] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:13.965624  2155 ts_tablet_manager.cc:1398] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.003s
I20240426 13:23:13.971215  2155 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:13.998140  2155 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:13.999935  2155 tablet_bootstrap.cc:492] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.000576  2155 ts_tablet_manager.cc:1367] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.007s
I20240426 13:23:14.003403  2155 raft_consensus.cc:348] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.003983  2155 raft_consensus.cc:729] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.005584  2155 consensus_queue.cc:260] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.007573  2155 ts_tablet_manager.cc:1398] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20240426 13:23:14.008270  2155 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.022099  2155 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.023043  2155 tablet_bootstrap.cc:492] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.023691  2155 ts_tablet_manager.cc:1367] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.000s
I20240426 13:23:14.025770  2155 raft_consensus.cc:348] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.026432  2155 raft_consensus.cc:729] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.026939  2155 consensus_queue.cc:260] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.028932  2155 ts_tablet_manager.cc:1398] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20240426 13:23:14.029618  2155 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.040835  2155 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.041594  2155 tablet_bootstrap.cc:492] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.042112  2155 ts_tablet_manager.cc:1367] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.002s
I20240426 13:23:14.043766  2155 raft_consensus.cc:348] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:14.044312  2155 raft_consensus.cc:729] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.044763  2155 consensus_queue.cc:260] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:14.046295  2155 ts_tablet_manager.cc:1398] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240426 13:23:14.047183  2155 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.061198  2155 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.061924  2155 tablet_bootstrap.cc:492] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.062484  2155 ts_tablet_manager.cc:1367] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20240426 13:23:14.064455  2155 raft_consensus.cc:348] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.065176  2155 raft_consensus.cc:729] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.065776  2155 consensus_queue.cc:260] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.067814  2155 ts_tablet_manager.cc:1398] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 13:23:14.069046  2155 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.081995  2155 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.082893  2155 tablet_bootstrap.cc:492] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.083554  2155 ts_tablet_manager.cc:1367] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.000s
I20240426 13:23:14.085659  2155 raft_consensus.cc:348] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.086292  2155 raft_consensus.cc:729] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.086906  2155 consensus_queue.cc:260] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.088996  2155 ts_tablet_manager.cc:1398] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.004s
I20240426 13:23:14.089689  2155 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.095876  2199 raft_consensus.cc:3037] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:14.108857  2155 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.109632  2155 tablet_bootstrap.cc:492] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.111145  2155 ts_tablet_manager.cc:1367] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
I20240426 13:23:14.113242  2155 raft_consensus.cc:348] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.113958  2155 raft_consensus.cc:729] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.114547  2155 consensus_queue.cc:260] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.116734  2155 ts_tablet_manager.cc:1398] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240426 13:23:14.117571  2155 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap starting.
I20240426 13:23:14.144261  2204 raft_consensus.cc:3037] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:14.157195  2155 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:23:14.158186  2155 tablet_bootstrap.cc:492] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Bootstrap complete.
I20240426 13:23:14.158808  2155 ts_tablet_manager.cc:1367] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent bootstrapping tablet: real 0.041s	user 0.014s	sys 0.023s
I20240426 13:23:14.161298  2155 raft_consensus.cc:348] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.162019  2155 raft_consensus.cc:729] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Initialized, Role: FOLLOWER
I20240426 13:23:14.162659  2155 consensus_queue.cc:260] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:14.164712  2155 ts_tablet_manager.cc:1398] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.003s
I20240426 13:23:14.245105  2250 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:23:14.245543  2250 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.246973  2250 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:14.247594  1473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf" is_pre_election: true
I20240426 13:23:14.247880  2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" is_pre_election: true
I20240426 13:23:14.248365  1473 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:14.248549  2204 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 1.
I20240426 13:23:14.249511  1515 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:14.250073  2250 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:23:14.250319  2250 raft_consensus.cc:482] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:23:14.250545  2250 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:14.255630  2250 raft_consensus.cc:504] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.257297  2250 leader_election.cc:288] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845)
I20240426 13:23:14.257907  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:14.258566  1476 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:14.258803  2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b813272c2184c4392e7f05b0011178f" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
I20240426 13:23:14.259404  2204 raft_consensus.cc:3037] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:23:14.265573  2204 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 3.
I20240426 13:23:14.265625  1476 raft_consensus.cc:2454] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 3.
I20240426 13:23:14.266427  1515 leader_election.cc:302] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:14.266959  2250 raft_consensus.cc:2781] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:14.267362  2250 raft_consensus.cc:686] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 3 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:14.268187  2250 consensus_queue.cc:237] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:14.277779  1298 catalog_manager.cc:5484] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 3, leader changed from 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 3 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:14.477047  2201 raft_consensus.cc:3037] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:14.707674  2201 raft_consensus.cc:1261] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:23:14.708989  2156 consensus_queue.cc:1035] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:23:14.732246  1473 raft_consensus.cc:1261] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:23:14.733801  2156 consensus_queue.cc:1035] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:21.774139 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:21.829963  1477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:21.830644  1477 raft_consensus.cc:595] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.857928  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33617d7b11304e939632cf23bc38f168"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:21.858510  1626 raft_consensus.cc:595] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.863447  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:21.863899  1626 raft_consensus.cc:595] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.880165  1552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a825e64679024315981ae5d0de008472"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:21.880770  1552 raft_consensus.cc:595] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.883489  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:21.884172  1626 raft_consensus.cc:595] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.886276  1477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:21.886689  1477 raft_consensus.cc:595] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:21.887585 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:23:21.888861 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:21.889534 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:21.906526  2358 raft_consensus.cc:979] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:21.907138  2363 raft_consensus.cc:1067] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:21.908325  2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a825e64679024315981ae5d0de008472"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:21.908776  2204 raft_consensus.cc:482] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:21.909018  2204 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.914515  2204 raft_consensus.cc:504] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:21.919576  2204 leader_election.cc:288] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:21.934511  2362 raft_consensus.cc:979] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:21.934988  2362 raft_consensus.cc:1067] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:21.934839  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:21.936074  1552 raft_consensus.cc:3032] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:21.936383  1552 raft_consensus.cc:729] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:21.937090  1552 consensus_queue.cc:260] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:21.938334  1552 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.938989  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43744
I20240426 13:23:21.939543  2201 raft_consensus.cc:482] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:21.939764  2201 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.941084  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a825e64679024315981ae5d0de008472" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:21.941706  1626 raft_consensus.cc:3037] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.944440  2201 raft_consensus.cc:504] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:21.945410  1552 raft_consensus.cc:2454] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:21.946864  2150 leader_election.cc:302] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:21.947940  2201 leader_election.cc:288] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:21.948377  1626 raft_consensus.cc:2454] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:21.948840  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:21.949491  1552 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.950469  2370 raft_consensus.cc:2781] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:21.955803  1552 raft_consensus.cc:2454] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:21.957518  2150 leader_election.cc:302] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:21.958678  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9a043f048f4bf4acb06f582821197b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:21.959303  2371 raft_consensus.cc:2781] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:21.958642  2370 raft_consensus.cc:686] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:21.959805  1477 raft_consensus.cc:3032] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:21.959877  2371 raft_consensus.cc:686] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:21.960633  1477 raft_consensus.cc:729] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:21.960500  2370 consensus_queue.cc:237] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:21.961201  2371 consensus_queue.cc:237] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:21.961740  1477 consensus_queue.cc:260] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:21.963603  1477 raft_consensus.cc:3037] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:21.971601  1477 raft_consensus.cc:2454] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:21.981839  1301 catalog_manager.cc:5484] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
W20240426 13:23:22.005810  2150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:33613, user_credentials={real_user=slave}} blocked reactor thread for 48397.7us
I20240426 13:23:22.018508  1301 catalog_manager.cc:5484] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:22.087477  2357 raft_consensus.cc:979] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.088004  2375 raft_consensus.cc:1067] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.089390  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43758
I20240426 13:23:22.090026  2201 raft_consensus.cc:482] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:22.090404  2201 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.097404  2201 raft_consensus.cc:504] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:22.099568  2201 leader_election.cc:288] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:22.099833  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:22.100445  1626 raft_consensus.cc:3032] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:22.100682  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2388dc8c06400baf4b699cfa351544" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:22.100754  1626 raft_consensus.cc:729] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:22.101344  1477 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.101744  1626 consensus_queue.cc:260] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:22.103060  1626 raft_consensus.cc:3037] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.107628  1477 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.109299  1626 raft_consensus.cc:2454] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.111238  2151 leader_election.cc:302] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:22.111989  2372 raft_consensus.cc:2781] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:22.112478  2372 raft_consensus.cc:686] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:22.113216  2372 consensus_queue.cc:237] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:22.122977  1301 catalog_manager.cc:5484] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:22.144337  2357 raft_consensus.cc:979] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.144810  2375 raft_consensus.cc:1067] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.146158  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33617d7b11304e939632cf23bc38f168"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43758
I20240426 13:23:22.146584  2201 raft_consensus.cc:482] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:22.146835  2201 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.151455  2201 raft_consensus.cc:504] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:22.153216  2201 leader_election.cc:288] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:22.153653  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:22.154254  1552 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.155905  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33617d7b11304e939632cf23bc38f168" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:22.156471  1626 raft_consensus.cc:3032] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:22.156682  1626 raft_consensus.cc:729] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:22.157222  1626 consensus_queue.cc:260] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:22.158253  1626 raft_consensus.cc:3037] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.159687  1552 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.160854  2150 leader_election.cc:302] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:22.161545  2372 raft_consensus.cc:2781] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:22.162016  2372 raft_consensus.cc:686] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:22.162829  2372 consensus_queue.cc:237] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:22.163993  1626 raft_consensus.cc:2454] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.171233  1301 catalog_manager.cc:5484] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:22.260066  2375 raft_consensus.cc:979] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.260569  2357 raft_consensus.cc:1067] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.261833  2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43758
I20240426 13:23:22.262253  2204 raft_consensus.cc:482] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:22.262486  2204 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.266755  2204 raft_consensus.cc:504] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:22.267972  2204 leader_election.cc:288] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:22.268712  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:22.268945  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c99cde390d4c2981806647821fd09b" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:22.269372  1552 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.269529  1622 raft_consensus.cc:3032] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:22.269914  1622 raft_consensus.cc:729] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:22.270612  1622 consensus_queue.cc:260] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:22.271865  1622 raft_consensus.cc:3037] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:22.275461  1552 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.276523  2150 leader_election.cc:302] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:22.277333  2372 raft_consensus.cc:2781] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:22.278676  2372 raft_consensus.cc:686] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:22.279448  2372 consensus_queue.cc:237] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:22.280747  1622 raft_consensus.cc:2454] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:22.289270  1301 catalog_manager.cc:5484] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:22.335034  2362 raft_consensus.cc:979] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.335539  2271 raft_consensus.cc:1067] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:22.336846  2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43744
I20240426 13:23:22.337540  2204 raft_consensus.cc:482] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:22.337868  2204 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:22.342446  2204 raft_consensus.cc:504] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:22.343717  2204 leader_election.cc:288] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:22.344894  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:22.345064  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb623ed697340f08d33b2d5e98eaedf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:22.345522  1476 raft_consensus.cc:3032] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:22.346944  1476 raft_consensus.cc:729] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:22.345547  1552 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:22.347680  1476 consensus_queue.cc:260] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:22.349130  1476 raft_consensus.cc:3037] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:22.355124  1552 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:22.356235  2150 leader_election.cc:302] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:22.357127  2372 raft_consensus.cc:2781] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:22.357563  2372 raft_consensus.cc:686] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:22.358338  2372 consensus_queue.cc:237] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:22.360149  1476 raft_consensus.cc:2454] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:22.368208  1301 catalog_manager.cc:5484] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 2 to 3, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 3 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:22.394634  1552 raft_consensus.cc:1261] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.396168  2372 consensus_queue.cc:1035] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.405537  1551 raft_consensus.cc:1261] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.407404  2371 consensus_queue.cc:1035] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:22.424762  1622 raft_consensus.cc:1261] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.427556  2371 consensus_queue.cc:1035] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.439409  1476 raft_consensus.cc:1261] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.442198  2372 consensus_queue.cc:1035] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.520139  1476 raft_consensus.cc:1261] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.521420  2372 consensus_queue.cc:1035] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.532328  1626 raft_consensus.cc:1261] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.534118  2371 consensus_queue.cc:1035] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.618863  1622 raft_consensus.cc:1261] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.619845  2371 consensus_queue.cc:1035] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.628338  1552 raft_consensus.cc:1261] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.629930  2388 consensus_queue.cc:1035] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.683853  1622 raft_consensus.cc:1261] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.684870  2370 consensus_queue.cc:1035] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.692778  1551 raft_consensus.cc:1261] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:22.696766  2372 consensus_queue.cc:1035] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:22.751521  1477 raft_consensus.cc:1261] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:22.752606  2372 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:22.760722  1551 raft_consensus.cc:1261] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:22.762307  2388 consensus_queue.cc:1035] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:22.938526 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:23.057143  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:23.058518  1626 raft_consensus.cc:595] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Received request to transfer leadership to TS 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:23.064947  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10594296adca430d8519c32089193198"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:23.065868  1626 raft_consensus.cc:595] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:23.073505  1552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:23.074208  1552 raft_consensus.cc:595] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:23.077836  1626 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:23.078655  1626 raft_consensus.cc:595] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Received request to transfer leadership to TS 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:23.081935  1552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e266712f988c42319d640fa890ec95d5"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:23.082752  1552 raft_consensus.cc:595] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:23.084575 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:23:23.087018 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:23.087841 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:23.197187  2395 raft_consensus.cc:979] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:23.197860  2375 raft_consensus.cc:1067] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Signalling peer 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:23.199242  1477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:39138
I20240426 13:23:23.199847  1477 raft_consensus.cc:482] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:23.200284  1477 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.209873  1477 raft_consensus.cc:504] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.214419  1477 leader_election.cc:288] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:23.215512  2395 raft_consensus.cc:979] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:23.216111  2395 raft_consensus.cc:1067] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Signalling peer 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:23.216099  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:23.216611  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:23.217489  1626 raft_consensus.cc:3032] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:23.217604  1552 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.217916  1626 raft_consensus.cc:729] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:23.218165  1477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:39138
I20240426 13:23:23.218920  1477 raft_consensus.cc:482] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:23.219318  1477 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.219163  1626 consensus_queue.cc:260] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.221056  1626 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.226356  1552 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:23.227753  1442 leader_election.cc:302] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:23.227793  1477 raft_consensus.cc:504] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.228788  2404 raft_consensus.cc:2781] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:23.229319  2404 raft_consensus.cc:686] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:23.230535  1477 leader_election.cc:288] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:23.230464  1626 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:23.230357  2404 consensus_queue.cc:237] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.232784  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:23.233662  1552 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.232903  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c56eac9ef7e45e39b06bab0ddc3275b" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:23.236214  1622 raft_consensus.cc:3032] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:23.236560  1622 raft_consensus.cc:729] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:23.237500  1622 consensus_queue.cc:260] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.238672  1622 raft_consensus.cc:3037] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.240775  1552 raft_consensus.cc:2454] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:23.247260  1442 leader_election.cc:302] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:23.248327  2271 raft_consensus.cc:2781] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:23.248850  2271 raft_consensus.cc:686] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:23.249724  2271 consensus_queue.cc:237] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:23.255291  1622 raft_consensus.cc:2454] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:23.261559  2363 raft_consensus.cc:979] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.262147  2405 raft_consensus.cc:1067] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.263401  1300 catalog_manager.cc:5484] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:23.264976  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:23.265453  2201 raft_consensus.cc:482] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:23.265704  2201 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.274622  2201 raft_consensus.cc:504] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.276638  2201 leader_election.cc:288] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:23.277666  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:23.278563  1552 raft_consensus.cc:3032] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:23.278926  1552 raft_consensus.cc:729] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:23.279908  1552 consensus_queue.cc:260] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.280473  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccda8d3014c4397a7e8987acdb33799" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:23.281311  1622 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.281677  1552 raft_consensus.cc:3037] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.288013  1622 raft_consensus.cc:2454] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:23.289234  2151 leader_election.cc:302] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:23.290006  1552 raft_consensus.cc:2454] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:23.290215  2371 raft_consensus.cc:2781] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:23.290797  2371 raft_consensus.cc:686] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:23.291839  2371 consensus_queue.cc:237] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.304476  1300 catalog_manager.cc:5484] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:23.305248  1301 catalog_manager.cc:5484] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:23.425496  2375 raft_consensus.cc:979] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.426191  2395 raft_consensus.cc:1067] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.427704  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10594296adca430d8519c32089193198"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43758
I20240426 13:23:23.428267  2201 raft_consensus.cc:482] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:23.428650  2201 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:23.436017  2201 raft_consensus.cc:504] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.438105  2201 leader_election.cc:288] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:23.439126  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:23.439779  1550 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:23.440259  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10594296adca430d8519c32089193198" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:23.440768  1622 raft_consensus.cc:3032] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:23.441051  1622 raft_consensus.cc:729] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:23.441740  1622 consensus_queue.cc:260] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.442782  1622 raft_consensus.cc:3037] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:23.447543  1550 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:23.448208  1622 raft_consensus.cc:2454] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:23.449148  2151 leader_election.cc:302] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:23.451181  2388 raft_consensus.cc:2781] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:23.451568  2388 raft_consensus.cc:686] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:23.452425  2388 consensus_queue.cc:237] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.462500  1301 catalog_manager.cc:5484] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 2 to 3, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 3 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:23.553478  2363 raft_consensus.cc:979] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.553855  2405 raft_consensus.cc:1067] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:23.555195  2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e266712f988c42319d640fa890ec95d5"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:23.555580  2204 raft_consensus.cc:482] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:23.555853  2204 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.562115  2204 raft_consensus.cc:504] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.563505  2204 leader_election.cc:288] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:23.564204  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:23.564402  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e266712f988c42319d640fa890ec95d5" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:23.564872  1550 raft_consensus.cc:3032] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:23.564931  1622 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.565176  1550 raft_consensus.cc:729] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:23.565904  1550 consensus_queue.cc:260] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.566891  1550 raft_consensus.cc:3037] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:23.570322  1622 raft_consensus.cc:2454] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:23.571339  2151 leader_election.cc:302] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:23.572117  2371 raft_consensus.cc:2781] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:23.572496  2371 raft_consensus.cc:686] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:23.572784  1550 raft_consensus.cc:2454] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:23.573236  2371 consensus_queue.cc:237] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:23.582408  1301 catalog_manager.cc:5484] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:23.671075  1626 raft_consensus.cc:1261] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.672065  2271 consensus_queue.cc:1035] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:23.684394  1550 raft_consensus.cc:1261] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.686852  2404 consensus_queue.cc:1035] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:23.702875  1550 raft_consensus.cc:1261] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.704005  2271 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:23.712378  1626 raft_consensus.cc:1261] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.714772  2362 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:23.759183  1550 raft_consensus.cc:1261] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.760305  2371 consensus_queue.cc:1035] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:23.772256  1626 raft_consensus.cc:1261] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:23.773406  2371 consensus_queue.cc:1035] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:23.861430  1626 raft_consensus.cc:1261] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:23.862509  2371 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:23.879339  1552 raft_consensus.cc:1261] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:23.881945  2427 consensus_queue.cc:1035] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:23:24.020179  1552 raft_consensus.cc:1261] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:24.021494  2426 consensus_queue.cc:1035] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:24.033995  1626 raft_consensus.cc:1261] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:24.037942  2388 consensus_queue.cc:1035] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:24.136454 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:24.172211  1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:24.172775  1550 raft_consensus.cc:595] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:24.175096  1477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80c15669d4654224aede733170c021f7"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:24.175643  1477 raft_consensus.cc:595] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:24.178121  1477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "948b43c3ee7f4e209c7c91300198700c"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:24.178503  1477 raft_consensus.cc:595] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Received request to transfer leadership to TS 948b43c3ee7f4e209c7c91300198700c
I20240426 13:23:24.180352  1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:24.180784  1550 raft_consensus.cc:595] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:24.182612  1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:24.183075  1550 raft_consensus.cc:595] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:24.184010 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:23:24.185626 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:24.186137 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:24.279341  2362 raft_consensus.cc:979] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower 948b43c3ee7f4e209c7c91300198700c to start an election
I20240426 13:23:24.279772  2432 raft_consensus.cc:1067] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Signalling peer 948b43c3ee7f4e209c7c91300198700c to start an election
I20240426 13:23:24.281225  1622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
 from {username='slave'} at 127.0.0.1:48890
I20240426 13:23:24.281754  1622 raft_consensus.cc:482] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:24.282021  1622 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:24.287050  1622 raft_consensus.cc:504] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.288995  1622 leader_election.cc:288] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 3 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:24.289734  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:24.289894  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8276c0795c4c4dab902ce8acda30340b" candidate_uuid: "948b43c3ee7f4e209c7c91300198700c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:24.290387  1477 raft_consensus.cc:3032] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:24.290565  1550 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:24.290736  1477 raft_consensus.cc:729] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:24.291549  1477 consensus_queue.cc:260] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.292749  1477 raft_consensus.cc:3037] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:24.296152  1550 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 3.
I20240426 13:23:24.297545  1590 leader_election.cc:302] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:24.298213  2395 raft_consensus.cc:2781] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:24.298717  2395 raft_consensus.cc:686] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 3 LEADER]: Becoming Leader. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:24.300153  2395 consensus_queue.cc:237] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.301201  1477 raft_consensus.cc:2454] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b43c3ee7f4e209c7c91300198700c in term 3.
I20240426 13:23:24.311247  1300 catalog_manager.cc:5484] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c reported cstate change: term changed from 2 to 3, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to 948b43c3ee7f4e209c7c91300198700c (127.23.61.4). New cstate: current_term: 3 leader_uuid: "948b43c3ee7f4e209c7c91300198700c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:24.461182  2432 raft_consensus.cc:979] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:24.461643  2404 raft_consensus.cc:1067] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:24.463063  2201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80c15669d4654224aede733170c021f7"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43744
I20240426 13:23:24.463608  2201 raft_consensus.cc:482] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:24.463925  2201 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.475149  2201 raft_consensus.cc:504] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:24.476676  2201 leader_election.cc:288] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:24.477535  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:24.478180  1477 raft_consensus.cc:3032] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:24.478214  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c15669d4654224aede733170c021f7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:24.478542  1477 raft_consensus.cc:729] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:24.478847  1622 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.479604  1477 consensus_queue.cc:260] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:24.480958  1477 raft_consensus.cc:3037] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.484597  1622 raft_consensus.cc:2454] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:24.487600  2151 leader_election.cc:302] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:24.488314  2388 raft_consensus.cc:2781] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:24.488714  2388 raft_consensus.cc:686] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:24.489531  2388 consensus_queue.cc:237] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:24.490402  1477 raft_consensus.cc:2454] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:24.499101  1300 catalog_manager.cc:5484] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:24.556912  2405 raft_consensus.cc:979] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:24.557281  2363 raft_consensus.cc:1067] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:24.558482  1477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:39142
I20240426 13:23:24.558916  1477 raft_consensus.cc:482] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:24.559154  1477 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.564702  1477 raft_consensus.cc:504] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.566908  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:24.567070  1477 leader_election.cc:288] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:24.567553  1552 raft_consensus.cc:3032] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:24.567868  1552 raft_consensus.cc:729] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:24.568006  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae5495d004b4fc0a78ef115fb4642f1" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:24.568703  1622 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.568722  1552 consensus_queue.cc:260] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.570396  1552 raft_consensus.cc:3037] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.575233  1622 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:24.576320  1443 leader_election.cc:302] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:24.576515  1552 raft_consensus.cc:2454] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:24.576927  2404 raft_consensus.cc:2781] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:24.577356  2404 raft_consensus.cc:686] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:24.578132  2404 consensus_queue.cc:237] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.587539  1300 catalog_manager.cc:5484] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:24.613505  2405 raft_consensus.cc:979] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:24.613983  2363 raft_consensus.cc:1067] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:24.615270  2204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:24.615717  2204 raft_consensus.cc:482] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:24.615981  2204 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.620946  2204 raft_consensus.cc:504] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:24.622210  2204 leader_election.cc:288] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:24.622931  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:24.623061  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a787774e5341acaa26b09a1a61388f" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:24.623514  1552 raft_consensus.cc:3032] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:24.623622  1622 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.623813  1552 raft_consensus.cc:729] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:24.624631  1552 consensus_queue.cc:260] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:24.625775  1552 raft_consensus.cc:3037] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.630842  1622 raft_consensus.cc:2454] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:24.631947  2151 leader_election.cc:302] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:24.632774  2426 raft_consensus.cc:2781] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:24.633169  2426 raft_consensus.cc:686] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:24.633927  2426 consensus_queue.cc:237] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:24.638307  1552 raft_consensus.cc:2454] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:24.643877  1300 catalog_manager.cc:5484] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:24.679772  1552 raft_consensus.cc:1261] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:24.681068  2428 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:24.681892  2405 raft_consensus.cc:979] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:24.682346  2363 raft_consensus.cc:1067] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:24.710827  1477 raft_consensus.cc:1261] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Refusing update from remote peer 948b43c3ee7f4e209c7c91300198700c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:24.710760  1476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:39142
I20240426 13:23:24.711547  1476 raft_consensus.cc:482] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:24.711948  1476 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.713996  2428 consensus_queue.cc:1035] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:24.722790  1476 raft_consensus.cc:504] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.730674  1476 leader_election.cc:288] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:24.731243  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:24.731906  1552 raft_consensus.cc:3032] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:24.732224  1552 raft_consensus.cc:729] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:24.735265  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:24.735848  1622 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.735280  1552 consensus_queue.cc:260] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.737358  1552 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:24.742444  1622 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:24.742919  1552 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 2.
I20240426 13:23:24.743669  1443 leader_election.cc:302] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:24.744486  2404 raft_consensus.cc:2781] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:24.744971  2404 raft_consensus.cc:686] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:24.745710  2404 consensus_queue.cc:237] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:24.756670  1300 catalog_manager.cc:5484] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 2 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:24.947983  1622 raft_consensus.cc:1261] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:24.949168  2426 consensus_queue.cc:1035] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:24.964432  1477 raft_consensus.cc:1261] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:24.966120  2426 consensus_queue.cc:1035] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.070662  1622 raft_consensus.cc:1261] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.071956  2404 consensus_queue.cc:1035] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.080103  1552 raft_consensus.cc:1261] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.081861  2404 consensus_queue.cc:1035] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.144148  1622 raft_consensus.cc:1261] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.145241  2446 consensus_queue.cc:1035] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.154760  1552 raft_consensus.cc:1261] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.156253  2388 consensus_queue.cc:1035] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.224411 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:25.264281  1473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51698a9fac6a431499553247279e5250"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:25.264817  1473 raft_consensus.cc:595] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Received request to transfer leadership to TS b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:25.266795  1473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d406b2568d574aedb189a69254c662aa"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:25.267244  1473 raft_consensus.cc:595] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Received request to transfer leadership to TS b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:25.269179  1622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0"
dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:44828
I20240426 13:23:25.269613  1622 raft_consensus.cc:595] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:25.270742 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 13:23:25.271891 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:25.272380 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:25.312132  2404 raft_consensus.cc:979] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:25.312739  2432 raft_consensus.cc:1067] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Signalling peer b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:25.314061  1552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d406b2568d574aedb189a69254c662aa"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:54366
I20240426 13:23:25.314632  1552 raft_consensus.cc:482] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:25.315014  1552 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.317592  1550 raft_consensus.cc:1261] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.318861  2432 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.321534  1552 raft_consensus.cc:504] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.323974  1552 leader_election.cc:288] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:25.324751  1473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:25.325373  1473 raft_consensus.cc:3032] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:25.325778  1473 raft_consensus.cc:729] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:25.326473  1473 consensus_queue.cc:260] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.327569  1473 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.333185  1625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d406b2568d574aedb189a69254c662aa" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:25.333824  1625 raft_consensus.cc:3037] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.336352  1626 raft_consensus.cc:1261] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.338932  2432 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.340219  1625 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:25.341935  1517 leader_election.cc:302] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:25.342667  2405 raft_consensus.cc:2781] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:25.343209  2405 raft_consensus.cc:686] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:25.344465  1473 raft_consensus.cc:2454] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:25.344265  2405 consensus_queue.cc:237] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.374686  1300 catalog_manager.cc:5484] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 2, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 2 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:25.447458  2448 raft_consensus.cc:979] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:25.447971  2448 raft_consensus.cc:1067] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Signalling peer b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:25.449609  1552 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51698a9fac6a431499553247279e5250"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:54366
I20240426 13:23:25.450055  1552 raft_consensus.cc:482] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:25.450304  1552 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.454735  1552 raft_consensus.cc:504] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.455982  1552 leader_election.cc:288] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:25.456724  1473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:25.456898  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:25.457242  1473 raft_consensus.cc:3032] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:25.457445  1622 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.457566  1473 raft_consensus.cc:729] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:25.458276  1473 consensus_queue.cc:260] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.459170  1473 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:25.461575  1622 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:25.462378  1517 leader_election.cc:302] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:25.462997  2456 raft_consensus.cc:2781] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:25.463394  2456 raft_consensus.cc:686] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:25.463734  1473 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 2.
I20240426 13:23:25.464325  2456 consensus_queue.cc:237] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:25.472432  1301 catalog_manager.cc:5484] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 1 to 2, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 2 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:25.592689  2375 raft_consensus.cc:979] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:25.593120  2395 raft_consensus.cc:1067] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:25.594543  2203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43758
I20240426 13:23:25.594976  2203 raft_consensus.cc:482] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:25.595407  2203 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:25.600436  2203 raft_consensus.cc:504] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:25.602010  2203 leader_election.cc:288] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Requested vote from peers 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539), 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131)
I20240426 13:23:25.603137  1473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:25.603575  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c8c0d5fc844f21b8ed5e96206020f0" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:25.604249  1622 raft_consensus.cc:3032] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:25.604548  1622 raft_consensus.cc:729] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 948b43c3ee7f4e209c7c91300198700c, State: Running, Role: LEADER
I20240426 13:23:25.603910  1473 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:25.605383  1622 consensus_queue.cc:260] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:25.606678  1622 raft_consensus.cc:3037] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:25.613481  1622 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:25.613525  1473 raft_consensus.cc:2454] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 3.
I20240426 13:23:25.614809  2149 leader_election.cc:302] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:25.615675  2388 raft_consensus.cc:2781] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:25.616202  2388 raft_consensus.cc:686] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:25.617061  2388 consensus_queue.cc:237] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } }
I20240426 13:23:25.625334  1301 catalog_manager.cc:5484] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 2 to 3, leader changed from 948b43c3ee7f4e209c7c91300198700c (127.23.61.4) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 3 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:25.750154  1473 raft_consensus.cc:1261] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.751379  2456 consensus_queue.cc:1035] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.761317  1622 raft_consensus.cc:1261] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.763393  2455 consensus_queue.cc:1035] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:23:25.867873  1473 raft_consensus.cc:1261] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.868929  2405 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:25.878965  1622 raft_consensus.cc:1261] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:25.881121  2363 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:26.021888  1473 raft_consensus.cc:1261] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:26.023263  2388 consensus_queue.cc:1035] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:26.036690  1622 raft_consensus.cc:1261] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:26.037796  2446 consensus_queue.cc:1035] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:26.305140 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:26.344702  1552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51698a9fac6a431499553247279e5250"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:26.345192  1552 raft_consensus.cc:595] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Received request to transfer leadership to TS 0d163e7cd1714dc48ebf662957edf6cf
I20240426 13:23:26.347254  1552 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:26.347790  1552 raft_consensus.cc:595] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:26.348695 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 13:23:26.349861 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:26.350342 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:26.446494  2363 raft_consensus.cc:979] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:26.447095  2405 raft_consensus.cc:1067] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Signalling peer 0d163e7cd1714dc48ebf662957edf6cf to start an election
I20240426 13:23:26.448618  1477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51698a9fac6a431499553247279e5250"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
 from {username='slave'} at 127.0.0.1:39142
I20240426 13:23:26.449163  1477 raft_consensus.cc:482] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:26.449534  1477 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:26.455056  1477 raft_consensus.cc:504] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:26.457396  1626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:26.457600  1552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51698a9fac6a431499553247279e5250" candidate_uuid: "0d163e7cd1714dc48ebf662957edf6cf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:26.457976  1626 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:26.458177  1552 raft_consensus.cc:3032] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:26.458492  1552 raft_consensus.cc:729] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:26.459218  1552 consensus_queue.cc:260] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:26.460183  1552 raft_consensus.cc:3037] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:26.464164  1477 leader_election.cc:288] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 3 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:26.465221  1626 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 3.
I20240426 13:23:26.465848  1622 raft_consensus.cc:1226] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Rejecting Update request from peer b156dff0c7b441c6b6f601006d0fcaea for earlier term 2. Current term is 3. Ops: []
I20240426 13:23:26.466542  1443 leader_election.cc:302] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:26.467397  2404 raft_consensus.cc:2781] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:26.467864  2404 raft_consensus.cc:686] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 LEADER]: Becoming Leader. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:26.468214  1552 raft_consensus.cc:2454] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d163e7cd1714dc48ebf662957edf6cf in term 3.
I20240426 13:23:26.468739  2404 consensus_queue.cc:237] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:26.477793  1300 catalog_manager.cc:5484] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf reported cstate change: term changed from 2 to 3, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2). New cstate: current_term: 3 leader_uuid: "0d163e7cd1714dc48ebf662957edf6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:26.745786  2455 raft_consensus.cc:979] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:26.746228  2363 raft_consensus.cc:1067] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:26.747442  2203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:26.747978  2203 raft_consensus.cc:482] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:26.748342  2203 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:26.754912  2203 raft_consensus.cc:504] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:26.756644  2203 leader_election.cc:288] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613)
I20240426 13:23:26.757309  1477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:26.757666  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e186fc847c7549b58d5a94639edf09cf" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:26.757954  1477 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:26.758293  1550 raft_consensus.cc:3032] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:26.758630  1550 raft_consensus.cc:729] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:26.759330  1550 consensus_queue.cc:260] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:26.760766  1550 raft_consensus.cc:3037] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:26.764816  1477 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:26.766000  2149 leader_election.cc:302] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d163e7cd1714dc48ebf662957edf6cf, 56e4ab2f7d204691a8f5a355a66a76a4; no voters: 
I20240426 13:23:26.767204  2446 raft_consensus.cc:2781] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:26.767876  2446 raft_consensus.cc:686] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:26.769625  1550 raft_consensus.cc:2454] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:26.769287  2446 consensus_queue.cc:237] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } }
I20240426 13:23:26.778337  1301 catalog_manager.cc:5484] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:27.026394  1626 raft_consensus.cc:1261] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:27.027504  2404 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:27.048913  1552 raft_consensus.cc:1261] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Refusing update from remote peer 0d163e7cd1714dc48ebf662957edf6cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:27.050371  2432 consensus_queue.cc:1035] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:27.343283  1552 raft_consensus.cc:1261] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:27.344399  2446 consensus_queue.cc:1035] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:27.349185  1477 raft_consensus.cc:1261] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:27.351065  2446 consensus_queue.cc:1035] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:27.388454 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:23:27.439723  2149 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:38131, user_credentials={real_user=slave}} blocked reactor thread for 66909.1us
I20240426 13:23:27.459056  1476 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190"
dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
mode: GRACEFUL
new_leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:57496
I20240426 13:23:27.459512  1476 raft_consensus.cc:595] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Received request to transfer leadership to TS b156dff0c7b441c6b6f601006d0fcaea
I20240426 13:23:27.460425 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:23:27.461686 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:27.462123 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:27.857625  2404 raft_consensus.cc:979] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: : Instructing follower b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:27.858103  2477 raft_consensus.cc:1067] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Signalling peer b156dff0c7b441c6b6f601006d0fcaea to start an election
I20240426 13:23:27.859426  1550 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
 from {username='slave'} at 127.0.0.1:54366
I20240426 13:23:27.859930  1550 raft_consensus.cc:482] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:27.860244  1550 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:27.865753  1550 raft_consensus.cc:504] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:27.867138  1550 leader_election.cc:288] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 election: Requested vote from peers 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:27.867928  1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0d163e7cd1714dc48ebf662957edf6cf"
I20240426 13:23:27.867937  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cadbce7a42d4cc181613ffd7ae24190" candidate_uuid: "b156dff0c7b441c6b6f601006d0fcaea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:27.868533  1476 raft_consensus.cc:3032] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:23:27.868628  1622 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:27.868856  1476 raft_consensus.cc:729] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d163e7cd1714dc48ebf662957edf6cf, State: Running, Role: LEADER
I20240426 13:23:27.869558  1476 consensus_queue.cc:260] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:27.870759  1476 raft_consensus.cc:3037] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:23:27.874532  1622 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 3.
I20240426 13:23:27.875514  1517 leader_election.cc:302] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b43c3ee7f4e209c7c91300198700c, b156dff0c7b441c6b6f601006d0fcaea; no voters: 
I20240426 13:23:27.876132  2455 raft_consensus.cc:2781] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:23:27.876576  2455 raft_consensus.cc:686] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 3 LEADER]: Becoming Leader. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:27.877282  2455 consensus_queue.cc:237] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } }
I20240426 13:23:27.880227  1476 raft_consensus.cc:2454] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b156dff0c7b441c6b6f601006d0fcaea in term 3.
I20240426 13:23:27.888291  1300 catalog_manager.cc:5484] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea reported cstate change: term changed from 2 to 3, leader changed from 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2) to b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3). New cstate: current_term: 3 leader_uuid: "b156dff0c7b441c6b6f601006d0fcaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } }
I20240426 13:23:28.356756  1622 raft_consensus.cc:1261] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:28.357717  2455 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:28.376763  1476 raft_consensus.cc:1261] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Refusing update from remote peer b156dff0c7b441c6b6f601006d0fcaea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:23:28.377940  2363 consensus_queue.cc:1035] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d163e7cd1714dc48ebf662957edf6cf" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:23:28.497326 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:28.530874  1550 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7"
dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
mode: GRACEFUL
new_leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:33556
I20240426 13:23:28.531347  1550 raft_consensus.cc:595] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Received request to transfer leadership to TS 56e4ab2f7d204691a8f5a355a66a76a4
I20240426 13:23:28.532351 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240426 13:23:28.533499 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:28.533953 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:29.068253  2493 raft_consensus.cc:979] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: : Instructing follower 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:29.068742  2363 raft_consensus.cc:1067] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Signalling peer 56e4ab2f7d204691a8f5a355a66a76a4 to start an election
I20240426 13:23:29.069876  2203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7"
dest_uuid: "56e4ab2f7d204691a8f5a355a66a76a4"
 from {username='slave'} at 127.0.0.1:43734
I20240426 13:23:29.070356  2203 raft_consensus.cc:482] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:23:29.070678  2203 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:29.077338  2203 raft_consensus.cc:504] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:29.078873  2203 leader_election.cc:288] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Requested vote from peers b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613), 948b43c3ee7f4e209c7c91300198700c (127.23.61.4:34539)
I20240426 13:23:29.079833  1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "948b43c3ee7f4e209c7c91300198700c"
I20240426 13:23:29.080107  1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b9b0be82342918f3a75e964df8da7" candidate_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b156dff0c7b441c6b6f601006d0fcaea"
I20240426 13:23:29.080441  1622 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:29.080725  1550 raft_consensus.cc:3032] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:23:29.081009  1550 raft_consensus.cc:729] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b156dff0c7b441c6b6f601006d0fcaea, State: Running, Role: LEADER
I20240426 13:23:29.081532  1550 consensus_queue.cc:260] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:29.082253  1550 raft_consensus.cc:3037] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:23:29.085440  1622 raft_consensus.cc:2454] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:29.086385  2151 leader_election.cc:302] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e4ab2f7d204691a8f5a355a66a76a4, 948b43c3ee7f4e209c7c91300198700c; no voters: 
I20240426 13:23:29.087131  2446 raft_consensus.cc:2781] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:23:29.087612  2446 raft_consensus.cc:686] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Becoming Leader. State: Replica: 56e4ab2f7d204691a8f5a355a66a76a4, State: Running, Role: LEADER
I20240426 13:23:29.088466  2446 consensus_queue.cc:237] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } }
I20240426 13:23:29.087559  1550 raft_consensus.cc:2454] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56e4ab2f7d204691a8f5a355a66a76a4 in term 2.
I20240426 13:23:29.099203  1300 catalog_manager.cc:5484] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 reported cstate change: term changed from 1 to 2, leader changed from b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3) to 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e4ab2f7d204691a8f5a355a66a76a4" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 43845 } health_report { overall_health: HEALTHY } } }
I20240426 13:23:29.532486  1622 raft_consensus.cc:1261] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:29.533413  2446 consensus_queue.cc:1035] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948b43c3ee7f4e209c7c91300198700c" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:29.537380  1550 raft_consensus.cc:1261] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Refusing update from remote peer 56e4ab2f7d204691a8f5a355a66a76a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:23:29.538574  2446 consensus_queue.cc:1035] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b156dff0c7b441c6b6f601006d0fcaea" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:23:29.567456 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:29.606021 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:23:29.607105 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:23:29.607539 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:29.643384 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:23:29.675201 23796 tablet_server.cc:178] TabletServer@127.23.61.1:43845 shutting down...
W20240426 13:23:29.700174  1515 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 70 similar messages]
W20240426 13:23:29.703984  1515 consensus_peers.cc:485] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.714807  1589 consensus_peers.cc:485] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.715624 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240426 13:23:29.715775  1589 consensus_peers.cc:485] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.716722 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.717494 23796 raft_consensus.cc:2229] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.717929 23796 raft_consensus.cc:2258] T 3b813272c2184c4392e7f05b0011178f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.720577 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.721503 23796 raft_consensus.cc:2229] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.722561 23796 raft_consensus.cc:2258] T 8c2388dc8c06400baf4b699cfa351544 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.725548 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.726562 23796 raft_consensus.cc:2229] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.727664 23796 raft_consensus.cc:2258] T 33617d7b11304e939632cf23bc38f168 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.730530 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.731288 23796 raft_consensus.cc:2229] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.731783 23796 raft_consensus.cc:2258] T 20dc8321cda547cc94f2dcdd71507e28 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.734670 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.735570 23796 raft_consensus.cc:2229] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.736490 23796 raft_consensus.cc:2258] T e266712f988c42319d640fa890ec95d5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.739367 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.741361 23796 raft_consensus.cc:2229] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.742071 23796 raft_consensus.cc:2258] T 191d125f2e474efabc176055155755b0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.746989 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.748188 23796 raft_consensus.cc:2229] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.748734 23796 raft_consensus.cc:2258] T 9bb1fddf692c4843acc3c5f994fdf53e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.752485 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.753605 23796 raft_consensus.cc:2229] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.754261 23796 raft_consensus.cc:2258] T 4b081604971346229e0323b88039b905 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.757043  1441 consensus_peers.cc:485] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.757310 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.758061 23796 raft_consensus.cc:2229] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.759011 23796 raft_consensus.cc:2258] T e186fc847c7549b58d5a94639edf09cf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.761732 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.762267 23796 raft_consensus.cc:2229] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.763021 23796 raft_consensus.cc:2258] T 2d0b9b0be82342918f3a75e964df8da7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.765939 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.766642 23796 raft_consensus.cc:2229] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.767499 23796 raft_consensus.cc:2258] T e8c8c0d5fc844f21b8ed5e96206020f0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.768769  1515 consensus_peers.cc:485] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.769975 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.770881 23796 raft_consensus.cc:2229] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.771271 23796 raft_consensus.cc:2258] T 8f7cffa351e841f2bbf2789cc31939cc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.773861 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.774363 23796 raft_consensus.cc:2229] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.774679 23796 raft_consensus.cc:2258] T 4efe65507a824cd88e8376761b8ebe1c P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.777004 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.777626 23796 raft_consensus.cc:2229] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.778023 23796 raft_consensus.cc:2258] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.780620 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.781304 23796 raft_consensus.cc:2229] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.781749 23796 raft_consensus.cc:2258] T 95477b67a3084e64a196d9d64957a3f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.784647 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.785228 23796 raft_consensus.cc:2229] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.785742 23796 raft_consensus.cc:2258] T 0013296c2f6d4bec82b9277e5df43e64 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.786242  1515 consensus_peers.cc:485] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.788188 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.788959 23796 raft_consensus.cc:2229] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.789633 23796 raft_consensus.cc:2258] T 10594296adca430d8519c32089193198 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.791651 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.792435 23796 raft_consensus.cc:2229] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.792793 23796 raft_consensus.cc:2258] T 3e3bd4a614b14384a7d4945594ab3871 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.795708 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.796540 23796 raft_consensus.cc:2229] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.797489 23796 raft_consensus.cc:2258] T df9a043f048f4bf4acb06f582821197b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.800308 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.800967 23796 raft_consensus.cc:2229] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.801443 23796 raft_consensus.cc:2258] T 0ca73bd7cd06413fbad853c7ccc749f2 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.803711 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.804543 23796 raft_consensus.cc:2229] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.804947 23796 raft_consensus.cc:2258] T c7af4930353e4432961278f8d6fc493e P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.806962 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.807518 23796 raft_consensus.cc:2229] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.807904 23796 raft_consensus.cc:2258] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.810058 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.810575 23796 raft_consensus.cc:2229] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.810972 23796 raft_consensus.cc:2258] T 8f73751c65ca4d568467edaf86391737 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.813597 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.814224 23796 raft_consensus.cc:2229] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.815348 23796 raft_consensus.cc:2258] T a825e64679024315981ae5d0de008472 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.816190  1589 consensus_peers.cc:485] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.817873 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.818507 23796 raft_consensus.cc:2229] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.818871 23796 raft_consensus.cc:2258] T 160c4150b51e426ab85b80343bc29efc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.820794 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.821301 23796 raft_consensus.cc:2229] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.822043 23796 raft_consensus.cc:2258] T b4c99cde390d4c2981806647821fd09b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.823966 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.824517 23796 raft_consensus.cc:2229] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.825210 23796 raft_consensus.cc:2258] T 3ccda8d3014c4397a7e8987acdb33799 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.827260 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.827898 23796 raft_consensus.cc:2229] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.828349 23796 raft_consensus.cc:2258] T 8c0863dde30e4fdab1de8f22d75fabd5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.829818  1441 consensus_peers.cc:485] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.830662 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.831302 23796 raft_consensus.cc:2229] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.831667 23796 raft_consensus.cc:2258] T 7e00f8df835646e89148ac3e3a2d6abc P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.833943 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.834653 23796 raft_consensus.cc:2229] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.835104 23796 raft_consensus.cc:2258] T a8ce3aa126ef422ca8fedc1e1349087b P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.835572  1441 consensus_peers.cc:485] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.837378 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.838006 23796 raft_consensus.cc:2229] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.838392 23796 raft_consensus.cc:2258] T b698eb433b2d49c8b17447ce4c5b28e5 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.840330 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.840802 23796 raft_consensus.cc:2229] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.841158 23796 raft_consensus.cc:2258] T 1ea2d370e87e4c90ad5c68256ea2b940 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.843072 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.843547 23796 raft_consensus.cc:2229] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.843892 23796 raft_consensus.cc:2258] T 8ffc860ae90a4860bc539207a9259897 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.845860 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.846387 23796 raft_consensus.cc:2229] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.846745 23796 raft_consensus.cc:2258] T 54bf83b19a6047a7bec69cda298d5436 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.848647 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.849148 23796 raft_consensus.cc:2229] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.849573 23796 raft_consensus.cc:2258] T 0e1fd3ed641c443095bc506c4c0107e7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.852073 23796 tablet_replica.cc:330] stopping tablet replica
W20240426 13:23:29.852002  1589 consensus_peers.cc:485] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.852922 23796 raft_consensus.cc:2229] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.853296 23796 raft_consensus.cc:2258] T 9766d6635658449298fdd0a3b0edae26 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.855126 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.855623 23796 raft_consensus.cc:2229] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.856017 23796 raft_consensus.cc:2258] T c2c76184e5b74fe4b94b24a896f5b89d P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.858364 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.858943 23796 raft_consensus.cc:2229] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.859377 23796 raft_consensus.cc:2258] T 134a02c7dd614ad8b152d809ebd56155 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.861806 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.862548 23796 raft_consensus.cc:2229] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.862928 23796 raft_consensus.cc:2258] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.865093 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.865588 23796 raft_consensus.cc:2229] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.865942 23796 raft_consensus.cc:2258] T f6b50872bb2142fe8a7a0c2e234d24af P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.867954 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.868536 23796 raft_consensus.cc:2229] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.869352 23796 raft_consensus.cc:2258] T d5a787774e5341acaa26b09a1a61388f P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.871569 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.872187 23796 raft_consensus.cc:2229] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.872607 23796 raft_consensus.cc:2258] T 612eff866f5b4a9e902f703f3f11eb82 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.874987 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.875612 23796 raft_consensus.cc:2229] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:29.876168 23796 raft_consensus.cc:2258] T 442e14b352624bdbb04e9a3fdc445d32 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.878501 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.879094 23796 raft_consensus.cc:2229] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.879899 23796 raft_consensus.cc:2258] T 0cb623ed697340f08d33b2d5e98eaedf P 56e4ab2f7d204691a8f5a355a66a76a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:29.882328 23796 tablet_replica.cc:330] stopping tablet replica
I20240426 13:23:29.882733 23796 raft_consensus.cc:2229] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:29.883419 23796 raft_consensus.cc:2258] T 80c15669d4654224aede733170c021f7 P 56e4ab2f7d204691a8f5a355a66a76a4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:29.889324  1515 consensus_peers.cc:485] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.890755  1515 consensus_peers.cc:485] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.907788  1589 consensus_peers.cc:485] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.918301  1515 consensus_peers.cc:485] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.926903  1441 consensus_peers.cc:485] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.954193  1589 consensus_peers.cc:485] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:29.959479 23796 tablet_server.cc:194] TabletServer@127.23.61.1:43845 shutdown complete.
W20240426 13:23:29.973327  1515 consensus_peers.cc:485] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:29.989965  1515 consensus_peers.cc:485] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.000193  1441 consensus_peers.cc:485] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.021744  1441 consensus_peers.cc:485] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.023531  1441 consensus_peers.cc:485] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.037015 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
W20240426 13:23:30.048020  1441 consensus_peers.cc:485] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.069500  1589 consensus_peers.cc:485] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.074303  1589 consensus_peers.cc:485] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.077445 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:23:30.078205 23796 tablet_replica.cc:330] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.079020 23796 raft_consensus.cc:2229] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.079571 23796 raft_consensus.cc:2258] T e8c8c0d5fc844f21b8ed5e96206020f0 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.082695 23796 tablet_replica.cc:330] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.083784 23796 raft_consensus.cc:2229] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.084365 23796 raft_consensus.cc:2258] T 6bc2512b5c104b73a15c3238bfedbe80 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.085441  1515 consensus_peers.cc:485] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.087260 23796 tablet_replica.cc:330] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.087913 23796 raft_consensus.cc:2229] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.088322 23796 raft_consensus.cc:2258] T 80c15669d4654224aede733170c021f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.090207 23796 tablet_replica.cc:330] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.090865 23796 raft_consensus.cc:2229] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.091790 23796 raft_consensus.cc:2258] T 3ae5495d004b4fc0a78ef115fb4642f1 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.094053 23796 tablet_replica.cc:330] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.094849 23796 raft_consensus.cc:2229] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.095381 23796 raft_consensus.cc:2258] T 8c2388dc8c06400baf4b699cfa351544 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.097736 23796 tablet_replica.cc:330] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.098376 23796 raft_consensus.cc:2229] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.099399 23796 raft_consensus.cc:2258] T 54bf83b19a6047a7bec69cda298d5436 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.101914 23796 tablet_replica.cc:330] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.102632 23796 raft_consensus.cc:2229] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.103134 23796 raft_consensus.cc:2258] T dc7a62fac3114417ae31d5ce8f6bbd86 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.105031 23796 tablet_replica.cc:330] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.105698 23796 raft_consensus.cc:2229] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.106130 23796 raft_consensus.cc:2258] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.108283 23796 tablet_replica.cc:330] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.108805 23796 raft_consensus.cc:2229] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.109524 23796 raft_consensus.cc:2258] T 4efe65507a824cd88e8376761b8ebe1c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.111403 23796 tablet_replica.cc:330] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.111974 23796 raft_consensus.cc:2229] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.112390 23796 raft_consensus.cc:2258] T 7e00f8df835646e89148ac3e3a2d6abc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.114466 23796 tablet_replica.cc:330] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.115316 23796 raft_consensus.cc:2229] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.115833 23796 raft_consensus.cc:2258] T a8ce3aa126ef422ca8fedc1e1349087b P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.117977 23796 tablet_replica.cc:330] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.118458 23796 raft_consensus.cc:2229] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.118867 23796 raft_consensus.cc:2258] T df9a043f048f4bf4acb06f582821197b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.121415 23796 tablet_replica.cc:330] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.122138 23796 raft_consensus.cc:2229] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.123018 23796 raft_consensus.cc:2258] T 3e3bd4a614b14384a7d4945594ab3871 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.123643  1589 consensus_peers.cc:485] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.125380 23796 tablet_replica.cc:330] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
W20240426 13:23:30.125952  1589 consensus_peers.cc:485] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.126282 23796 raft_consensus.cc:2229] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
W20240426 13:23:30.126821  1515 consensus_peers.cc:485] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.127382 23796 raft_consensus.cc:2258] T 95477b67a3084e64a196d9d64957a3f7 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.129464 23796 tablet_replica.cc:330] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.129997 23796 raft_consensus.cc:2229] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.130395 23796 raft_consensus.cc:2258] T 919c4656a25c42e6a02d8d0a8d1f7f59 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.132345 23796 tablet_replica.cc:330] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.132884 23796 raft_consensus.cc:2229] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.133287 23796 raft_consensus.cc:2258] T 0cb623ed697340f08d33b2d5e98eaedf P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.135430 23796 tablet_replica.cc:330] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.136207 23796 raft_consensus.cc:2229] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.137193 23796 raft_consensus.cc:2258] T 289eab6dc655403aa7b8a801972d0eaf P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.140600 23796 tablet_replica.cc:330] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.141332 23796 raft_consensus.cc:2229] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.141767 23796 raft_consensus.cc:2258] T 8276c0795c4c4dab902ce8acda30340b P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.144423 23796 tablet_replica.cc:330] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.145164 23796 raft_consensus.cc:2229] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.145732 23796 raft_consensus.cc:2258] T 612eff866f5b4a9e902f703f3f11eb82 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.147147  1589 consensus_peers.cc:485] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.148366 23796 tablet_replica.cc:330] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.149292 23796 raft_consensus.cc:2229] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.150014 23796 raft_consensus.cc:2258] T 51698a9fac6a431499553247279e5250 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.152412 23796 tablet_replica.cc:330] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.153092 23796 raft_consensus.cc:2229] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.153610 23796 raft_consensus.cc:2258] T 8f7cffa351e841f2bbf2789cc31939cc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.156189 23796 tablet_replica.cc:330] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.156972 23796 raft_consensus.cc:2229] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.157943 23796 raft_consensus.cc:2258] T f6b50872bb2142fe8a7a0c2e234d24af P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.159001  1589 consensus_peers.cc:485] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.159435  1589 consensus_peers.cc:485] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.160485 23796 tablet_replica.cc:330] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.161316 23796 raft_consensus.cc:2229] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.161834 23796 raft_consensus.cc:2258] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.164484 23796 tablet_replica.cc:330] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.165218 23796 raft_consensus.cc:2229] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.165702 23796 raft_consensus.cc:2258] T 9cadbce7a42d4cc181613ffd7ae24190 P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.168188 23796 tablet_replica.cc:330] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.168974 23796 raft_consensus.cc:2229] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.169584 23796 raft_consensus.cc:2258] T 442e14b352624bdbb04e9a3fdc445d32 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.171970 23796 tablet_replica.cc:330] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.172569 23796 raft_consensus.cc:2229] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.172958 23796 raft_consensus.cc:2258] T f8cbfde04d874c1bb4aac75641436065 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.175163 23796 tablet_replica.cc:330] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.175839 23796 raft_consensus.cc:2229] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.176784 23796 raft_consensus.cc:2258] T 20dc8321cda547cc94f2dcdd71507e28 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.178689  1515 consensus_peers.cc:485] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.179217 23796 tablet_replica.cc:330] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.179791 23796 raft_consensus.cc:2229] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.180191 23796 raft_consensus.cc:2258] T 3b813272c2184c4392e7f05b0011178f P 0d163e7cd1714dc48ebf662957edf6cf [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.182132 23796 tablet_replica.cc:330] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.182624 23796 raft_consensus.cc:2229] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.183322 23796 raft_consensus.cc:2258] T 0c56eac9ef7e45e39b06bab0ddc3275b P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.185180 23796 tablet_replica.cc:330] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.185679 23796 raft_consensus.cc:2229] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.186017 23796 raft_consensus.cc:2258] T d406b2568d574aedb189a69254c662aa P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.188009 23796 tablet_replica.cc:330] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.188673 23796 raft_consensus.cc:2229] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.189170 23796 raft_consensus.cc:2258] T 9bb1fddf692c4843acc3c5f994fdf53e P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.191242  1515 consensus_peers.cc:485] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.191730 23796 tablet_replica.cc:330] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.192353 23796 raft_consensus.cc:2229] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.192725 23796 raft_consensus.cc:2258] T 191d125f2e474efabc176055155755b0 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.194677 23796 tablet_replica.cc:330] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.195221 23796 raft_consensus.cc:2229] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.195662 23796 raft_consensus.cc:2258] T 074e277f90624d9e8754dfd0a7bb703d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.197427 23796 tablet_replica.cc:330] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.197886 23796 raft_consensus.cc:2229] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.198609 23796 raft_consensus.cc:2258] T 160c4150b51e426ab85b80343bc29efc P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.200484 23796 tablet_replica.cc:330] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.201081 23796 raft_consensus.cc:2229] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.201594 23796 raft_consensus.cc:2258] T acd95475394f4c5bb64e94326a4cf9f5 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.204205 23796 tablet_replica.cc:330] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
W20240426 13:23:30.204747  1515 consensus_peers.cc:485] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.204852 23796 raft_consensus.cc:2229] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.206158 23796 raft_consensus.cc:2258] T 9766d6635658449298fdd0a3b0edae26 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.208513 23796 tablet_replica.cc:330] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
W20240426 13:23:30.208511  1515 consensus_peers.cc:485] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.209379 23796 raft_consensus.cc:2229] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.209815 23796 raft_consensus.cc:2258] T 0e1fd3ed641c443095bc506c4c0107e7 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.211719 23796 tablet_replica.cc:330] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.212241 23796 raft_consensus.cc:2229] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.212937 23796 raft_consensus.cc:2258] T c2c76184e5b74fe4b94b24a896f5b89d P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.214731 23796 tablet_replica.cc:330] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.215214 23796 raft_consensus.cc:2229] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.215628 23796 raft_consensus.cc:2258] T 1ea2d370e87e4c90ad5c68256ea2b940 P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.217485 23796 tablet_replica.cc:330] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.217958 23796 raft_consensus.cc:2229] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.218648 23796 raft_consensus.cc:2258] T 4b081604971346229e0323b88039b905 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.220469 23796 tablet_replica.cc:330] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.220927 23796 raft_consensus.cc:2229] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.221330 23796 raft_consensus.cc:2258] T e186fc847c7549b58d5a94639edf09cf P 0d163e7cd1714dc48ebf662957edf6cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.223181 23796 tablet_replica.cc:330] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.223635 23796 raft_consensus.cc:2229] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.224560 23796 raft_consensus.cc:2258] T eb745ab6644f4cfb9c10676dadd72f29 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.226210  1589 consensus_peers.cc:485] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.226970 23796 tablet_replica.cc:330] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.227458 23796 raft_consensus.cc:2229] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.227833 23796 raft_consensus.cc:2258] T f376ec882d7545d8bb64b314b89dc70c P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.229645 23796 tablet_replica.cc:330] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf: stopping tablet replica
I20240426 13:23:30.230064 23796 raft_consensus.cc:2229] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.230429 23796 raft_consensus.cc:2258] T 0013296c2f6d4bec82b9277e5df43e64 P 0d163e7cd1714dc48ebf662957edf6cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.244735  1515 consensus_peers.cc:485] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 56e4ab2f7d204691a8f5a355a66a76a4 (127.23.61.1:43845): Couldn't send request to peer 56e4ab2f7d204691a8f5a355a66a76a4. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:43845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.245038  1589 consensus_peers.cc:485] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.250388  1515 consensus_peers.cc:485] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.304560  1515 consensus_peers.cc:485] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.305565 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
W20240426 13:23:30.306684  1589 consensus_peers.cc:485] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.313544  1589 consensus_peers.cc:485] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.322700  1589 consensus_peers.cc:485] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.345995  1589 consensus_peers.cc:485] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.371119 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
W20240426 13:23:30.376302  1589 consensus_peers.cc:485] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.418941  1515 consensus_peers.cc:485] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.420910 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:23:30.421831 23796 tablet_replica.cc:330] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.422735 23796 raft_consensus.cc:2229] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.423308 23796 raft_consensus.cc:2258] T 9766d6635658449298fdd0a3b0edae26 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.425522 23796 tablet_replica.cc:330] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.426044 23796 raft_consensus.cc:2229] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.426481 23796 raft_consensus.cc:2258] T 3ae5495d004b4fc0a78ef115fb4642f1 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.428653 23796 tablet_replica.cc:330] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.429157 23796 raft_consensus.cc:2229] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
W20240426 13:23:30.429208  1590 consensus_peers.cc:485] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.429555 23796 raft_consensus.cc:2258] T 4efe65507a824cd88e8376761b8ebe1c P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.431373 23796 tablet_replica.cc:330] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.431893 23796 raft_consensus.cc:2229] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.432293 23796 raft_consensus.cc:2258] T 6bc2512b5c104b73a15c3238bfedbe80 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.434021 23796 tablet_replica.cc:330] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.434542 23796 raft_consensus.cc:2229] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.435295 23796 raft_consensus.cc:2258] T 0ca73bd7cd06413fbad853c7ccc749f2 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.437796 23796 tablet_replica.cc:330] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.438542 23796 raft_consensus.cc:2229] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.438966 23796 raft_consensus.cc:2258] T df9a043f048f4bf4acb06f582821197b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.441345 23796 tablet_replica.cc:330] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.442075 23796 raft_consensus.cc:2229] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.442459 23796 raft_consensus.cc:2258] T 10594296adca430d8519c32089193198 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.445201 23796 tablet_replica.cc:330] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.445910 23796 raft_consensus.cc:2229] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.446609 23796 raft_consensus.cc:2258] T a8ce3aa126ef422ca8fedc1e1349087b P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.448678 23796 tablet_replica.cc:330] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.449191 23796 raft_consensus.cc:2229] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.449538 23796 raft_consensus.cc:2258] T 51698a9fac6a431499553247279e5250 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.451236 23796 tablet_replica.cc:330] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.451745 23796 raft_consensus.cc:2229] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.452180 23796 raft_consensus.cc:2258] T 0cb623ed697340f08d33b2d5e98eaedf P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.454316 23796 tablet_replica.cc:330] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.454885 23796 raft_consensus.cc:2229] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.455415 23796 raft_consensus.cc:2258] T 289eab6dc655403aa7b8a801972d0eaf P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.456808  1589 consensus_peers.cc:485] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.457979 23796 tablet_replica.cc:330] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.458779 23796 raft_consensus.cc:2229] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
W20240426 13:23:30.459038  1515 consensus_peers.cc:485] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.459235 23796 raft_consensus.cc:2258] T f6b50872bb2142fe8a7a0c2e234d24af P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.461148 23796 tablet_replica.cc:330] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.461634 23796 raft_consensus.cc:2229] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.462327 23796 raft_consensus.cc:2258] T 134a02c7dd614ad8b152d809ebd56155 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.464010 23796 tablet_replica.cc:330] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.464541 23796 raft_consensus.cc:2229] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.465200 23796 raft_consensus.cc:2258] T 9cadbce7a42d4cc181613ffd7ae24190 P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.466864 23796 tablet_replica.cc:330] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.467301 23796 raft_consensus.cc:2229] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.467635 23796 raft_consensus.cc:2258] T a825e64679024315981ae5d0de008472 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.469326 23796 tablet_replica.cc:330] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.469759 23796 raft_consensus.cc:2229] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.470454 23796 raft_consensus.cc:2258] T 8f73751c65ca4d568467edaf86391737 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.472733 23796 tablet_replica.cc:330] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.473402 23796 raft_consensus.cc:2229] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.474292 23796 raft_consensus.cc:2258] T 3b813272c2184c4392e7f05b0011178f P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.476730 23796 tablet_replica.cc:330] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.477201 23796 raft_consensus.cc:2229] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.477540 23796 raft_consensus.cc:2258] T 0c56eac9ef7e45e39b06bab0ddc3275b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.479279 23796 tablet_replica.cc:330] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.479746 23796 raft_consensus.cc:2229] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.480149 23796 raft_consensus.cc:2258] T b4c99cde390d4c2981806647821fd09b P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.481774 23796 tablet_replica.cc:330] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.482321 23796 raft_consensus.cc:2229] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.482676 23796 raft_consensus.cc:2258] T e266712f988c42319d640fa890ec95d5 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.484375 23796 tablet_replica.cc:330] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.484831 23796 raft_consensus.cc:2229] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.485546 23796 raft_consensus.cc:2258] T 191d125f2e474efabc176055155755b0 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.487351 23796 tablet_replica.cc:330] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.488018 23796 raft_consensus.cc:2229] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.488893 23796 raft_consensus.cc:2258] T 9bb1fddf692c4843acc3c5f994fdf53e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.491372 23796 tablet_replica.cc:330] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.492167 23796 raft_consensus.cc:2229] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.492523 23796 raft_consensus.cc:2258] T 8276c0795c4c4dab902ce8acda30340b P b156dff0c7b441c6b6f601006d0fcaea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.494177 23796 tablet_replica.cc:330] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.494863 23796 raft_consensus.cc:2229] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.495628 23796 raft_consensus.cc:2258] T 612eff866f5b4a9e902f703f3f11eb82 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.497473 23796 tablet_replica.cc:330] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.498006 23796 raft_consensus.cc:2229] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.498396 23796 raft_consensus.cc:2258] T d5a787774e5341acaa26b09a1a61388f P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.500243 23796 tablet_replica.cc:330] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.500928 23796 raft_consensus.cc:2229] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.501720 23796 raft_consensus.cc:2258] T d406b2568d574aedb189a69254c662aa P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.503351 23796 tablet_replica.cc:330] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.503813 23796 raft_consensus.cc:2229] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.504554 23796 raft_consensus.cc:2258] T 074e277f90624d9e8754dfd0a7bb703d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.506605 23796 tablet_replica.cc:330] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.507071 23796 raft_consensus.cc:2229] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.507407 23796 raft_consensus.cc:2258] T c2c76184e5b74fe4b94b24a896f5b89d P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.509404 23796 tablet_replica.cc:330] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.509862 23796 raft_consensus.cc:2229] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.510630 23796 raft_consensus.cc:2258] T 8c0863dde30e4fdab1de8f22d75fabd5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.511314  1590 consensus_peers.cc:485] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.512365 23796 tablet_replica.cc:330] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.512988 23796 raft_consensus.cc:2229] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.513408 23796 raft_consensus.cc:2258] T 3ccda8d3014c4397a7e8987acdb33799 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.515688 23796 tablet_replica.cc:330] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.516399 23796 raft_consensus.cc:2229] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.516868 23796 raft_consensus.cc:2258] T b698eb433b2d49c8b17447ce4c5b28e5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.518810 23796 tablet_replica.cc:330] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.519268 23796 raft_consensus.cc:2229] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.519610 23796 raft_consensus.cc:2258] T dc7a62fac3114417ae31d5ce8f6bbd86 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.521273 23796 tablet_replica.cc:330] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.521709 23796 raft_consensus.cc:2229] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.522063 23796 raft_consensus.cc:2258] T 8ffc860ae90a4860bc539207a9259897 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.523617 23796 tablet_replica.cc:330] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.524147 23796 raft_consensus.cc:2229] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.524493 23796 raft_consensus.cc:2258] T 4b081604971346229e0323b88039b905 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.526257 23796 tablet_replica.cc:330] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.526710 23796 raft_consensus.cc:2229] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.527101 23796 raft_consensus.cc:2258] T 2d0b9b0be82342918f3a75e964df8da7 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.529708 23796 tablet_replica.cc:330] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.530164 23796 raft_consensus.cc:2229] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.530500 23796 raft_consensus.cc:2258] T e186fc847c7549b58d5a94639edf09cf P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.532240 23796 tablet_replica.cc:330] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.532673 23796 raft_consensus.cc:2229] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.533346 23796 raft_consensus.cc:2258] T c7af4930353e4432961278f8d6fc493e P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.535030 23796 tablet_replica.cc:330] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.535492 23796 raft_consensus.cc:2229] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.536249 23796 raft_consensus.cc:2258] T 919c4656a25c42e6a02d8d0a8d1f7f59 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.537966 23796 tablet_replica.cc:330] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.538403 23796 raft_consensus.cc:2229] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.538746 23796 raft_consensus.cc:2258] T eb745ab6644f4cfb9c10676dadd72f29 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.540385 23796 tablet_replica.cc:330] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.540805 23796 raft_consensus.cc:2229] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.541558 23796 raft_consensus.cc:2258] T f376ec882d7545d8bb64b314b89dc70c P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.544229 23796 tablet_replica.cc:330] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.544869 23796 raft_consensus.cc:2229] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.545349 23796 raft_consensus.cc:2258] T f8cbfde04d874c1bb4aac75641436065 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.547930 23796 tablet_replica.cc:330] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.548581 23796 raft_consensus.cc:2229] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.549057 23796 raft_consensus.cc:2258] T 20dc8321cda547cc94f2dcdd71507e28 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.551645 23796 tablet_replica.cc:330] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.554736 23796 raft_consensus.cc:2229] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.555233 23796 raft_consensus.cc:2258] T 33617d7b11304e939632cf23bc38f168 P b156dff0c7b441c6b6f601006d0fcaea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.558466 23796 tablet_replica.cc:330] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea: stopping tablet replica
I20240426 13:23:30.558890 23796 raft_consensus.cc:2229] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus shutting down.
W20240426 13:23:30.558890  1589 consensus_peers.cc:485] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.559406 23796 raft_consensus.cc:2258] T acd95475394f4c5bb64e94326a4cf9f5 P b156dff0c7b441c6b6f601006d0fcaea [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:23:30.584506  1589 consensus_peers.cc:485] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c -> Peer 0d163e7cd1714dc48ebf662957edf6cf (127.23.61.2:38131): Couldn't send request to peer 0d163e7cd1714dc48ebf662957edf6cf. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:38131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:23:30.618746  1590 consensus_peers.cc:485] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.3:33613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.634204 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
W20240426 13:23:30.670450  1590 consensus_peers.cc:485] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.3:33613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.708947 23796 tablet_server.cc:178] TabletServer@127.23.61.4:0 shutting down...
W20240426 13:23:30.725016  1590 consensus_peers.cc:485] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c -> Peer b156dff0c7b441c6b6f601006d0fcaea (127.23.61.3:33613): Couldn't send request to peer b156dff0c7b441c6b6f601006d0fcaea. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.3:33613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:23:30.749826 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:23:30.750690 23796 tablet_replica.cc:330] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.751714 23796 raft_consensus.cc:2229] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.752346 23796 raft_consensus.cc:2258] T c7af4930353e4432961278f8d6fc493e P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.754488 23796 tablet_replica.cc:330] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.755050 23796 raft_consensus.cc:2229] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.755424 23796 raft_consensus.cc:2258] T e8c8c0d5fc844f21b8ed5e96206020f0 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.757241 23796 tablet_replica.cc:330] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.757795 23796 raft_consensus.cc:2229] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.758219 23796 raft_consensus.cc:2258] T 80c15669d4654224aede733170c021f7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.760550 23796 tablet_replica.cc:330] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.761284 23796 raft_consensus.cc:2229] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.761746 23796 raft_consensus.cc:2258] T e266712f988c42319d640fa890ec95d5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.763777 23796 tablet_replica.cc:330] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.764324 23796 raft_consensus.cc:2229] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.765013 23796 raft_consensus.cc:2258] T 6bc2512b5c104b73a15c3238bfedbe80 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.766738 23796 tablet_replica.cc:330] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.767208 23796 raft_consensus.cc:2229] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.767513 23796 raft_consensus.cc:2258] T 3ae5495d004b4fc0a78ef115fb4642f1 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.769169 23796 tablet_replica.cc:330] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.769634 23796 raft_consensus.cc:2229] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.769958 23796 raft_consensus.cc:2258] T 10594296adca430d8519c32089193198 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.771581 23796 tablet_replica.cc:330] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.772076 23796 raft_consensus.cc:2229] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.772398 23796 raft_consensus.cc:2258] T 8c2388dc8c06400baf4b699cfa351544 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.774055 23796 tablet_replica.cc:330] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.774489 23796 raft_consensus.cc:2229] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.774816 23796 raft_consensus.cc:2258] T 2d0b9b0be82342918f3a75e964df8da7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.776439 23796 tablet_replica.cc:330] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.776898 23796 raft_consensus.cc:2229] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.777575 23796 raft_consensus.cc:2258] T 8276c0795c4c4dab902ce8acda30340b P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.779170 23796 tablet_replica.cc:330] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.779618 23796 raft_consensus.cc:2229] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.780341 23796 raft_consensus.cc:2258] T f5eec2b6b55d4310bfd18eb2f7b093a5 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.781957 23796 tablet_replica.cc:330] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.782402 23796 raft_consensus.cc:2229] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.782701 23796 raft_consensus.cc:2258] T 0ca73bd7cd06413fbad853c7ccc749f2 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.784384 23796 tablet_replica.cc:330] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.784835 23796 raft_consensus.cc:2229] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.785156 23796 raft_consensus.cc:2258] T 3e3bd4a614b14384a7d4945594ab3871 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.786811 23796 tablet_replica.cc:330] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.787256 23796 raft_consensus.cc:2229] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.787596 23796 raft_consensus.cc:2258] T 289eab6dc655403aa7b8a801972d0eaf P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.789315 23796 tablet_replica.cc:330] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.789736 23796 raft_consensus.cc:2229] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.790107 23796 raft_consensus.cc:2258] T 95477b67a3084e64a196d9d64957a3f7 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.791858 23796 tablet_replica.cc:330] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.792346 23796 raft_consensus.cc:2229] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.792690 23796 raft_consensus.cc:2258] T 51698a9fac6a431499553247279e5250 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.794337 23796 tablet_replica.cc:330] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.794785 23796 raft_consensus.cc:2229] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.795516 23796 raft_consensus.cc:2258] T 8f7cffa351e841f2bbf2789cc31939cc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.797344 23796 tablet_replica.cc:330] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.797819 23796 raft_consensus.cc:2229] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.798189 23796 raft_consensus.cc:2258] T a825e64679024315981ae5d0de008472 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.799818 23796 tablet_replica.cc:330] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.800305 23796 raft_consensus.cc:2229] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.800653 23796 raft_consensus.cc:2258] T 160c4150b51e426ab85b80343bc29efc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.802400 23796 tablet_replica.cc:330] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.802858 23796 raft_consensus.cc:2229] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.803295 23796 raft_consensus.cc:2258] T 8f73751c65ca4d568467edaf86391737 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.805007 23796 tablet_replica.cc:330] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.805512 23796 raft_consensus.cc:2229] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.805868 23796 raft_consensus.cc:2258] T 134a02c7dd614ad8b152d809ebd56155 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.807607 23796 tablet_replica.cc:330] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.808084 23796 raft_consensus.cc:2229] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.808729 23796 raft_consensus.cc:2258] T 7b84898ee62e4f8eb0fb983c9c927fe0 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.810397 23796 tablet_replica.cc:330] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.810814 23796 raft_consensus.cc:2229] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.811164 23796 raft_consensus.cc:2258] T 9cadbce7a42d4cc181613ffd7ae24190 P 948b43c3ee7f4e209c7c91300198700c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.812839 23796 tablet_replica.cc:330] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.813338 23796 raft_consensus.cc:2229] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.813725 23796 raft_consensus.cc:2258] T d5a787774e5341acaa26b09a1a61388f P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.815668 23796 tablet_replica.cc:330] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.816236 23796 raft_consensus.cc:2229] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.816938 23796 raft_consensus.cc:2258] T f8cbfde04d874c1bb4aac75641436065 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.818755 23796 tablet_replica.cc:330] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.819233 23796 raft_consensus.cc:2229] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.819653 23796 raft_consensus.cc:2258] T 33617d7b11304e939632cf23bc38f168 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.821473 23796 tablet_replica.cc:330] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.821966 23796 raft_consensus.cc:2229] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.822713 23796 raft_consensus.cc:2258] T acd95475394f4c5bb64e94326a4cf9f5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.824582 23796 tablet_replica.cc:330] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.825079 23796 raft_consensus.cc:2229] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.825801 23796 raft_consensus.cc:2258] T 0e1fd3ed641c443095bc506c4c0107e7 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.827626 23796 tablet_replica.cc:330] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.828151 23796 raft_consensus.cc:2229] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.828501 23796 raft_consensus.cc:2258] T d406b2568d574aedb189a69254c662aa P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.830164 23796 tablet_replica.cc:330] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.830626 23796 raft_consensus.cc:2229] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.831028 23796 raft_consensus.cc:2258] T 074e277f90624d9e8754dfd0a7bb703d P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.832856 23796 tablet_replica.cc:330] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.833300 23796 raft_consensus.cc:2229] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.833678 23796 raft_consensus.cc:2258] T eb745ab6644f4cfb9c10676dadd72f29 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.835340 23796 tablet_replica.cc:330] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.835842 23796 raft_consensus.cc:2229] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.836237 23796 raft_consensus.cc:2258] T f376ec882d7545d8bb64b314b89dc70c P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.838047 23796 tablet_replica.cc:330] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.838512 23796 raft_consensus.cc:2229] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.838868 23796 raft_consensus.cc:2258] T 0c56eac9ef7e45e39b06bab0ddc3275b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.840593 23796 tablet_replica.cc:330] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.841063 23796 raft_consensus.cc:2229] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.841758 23796 raft_consensus.cc:2258] T 442e14b352624bdbb04e9a3fdc445d32 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.843605 23796 tablet_replica.cc:330] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.844152 23796 raft_consensus.cc:2229] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.844520 23796 raft_consensus.cc:2258] T 8c0863dde30e4fdab1de8f22d75fabd5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.846235 23796 tablet_replica.cc:330] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.846688 23796 raft_consensus.cc:2229] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.847381 23796 raft_consensus.cc:2258] T 7e00f8df835646e89148ac3e3a2d6abc P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.849283 23796 tablet_replica.cc:330] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.849771 23796 raft_consensus.cc:2229] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.850137 23796 raft_consensus.cc:2258] T 3ccda8d3014c4397a7e8987acdb33799 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.851729 23796 tablet_replica.cc:330] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.852200 23796 raft_consensus.cc:2229] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.852849 23796 raft_consensus.cc:2258] T 1ea2d370e87e4c90ad5c68256ea2b940 P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.854593 23796 tablet_replica.cc:330] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.855098 23796 raft_consensus.cc:2229] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.855859 23796 raft_consensus.cc:2258] T b698eb433b2d49c8b17447ce4c5b28e5 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.857694 23796 tablet_replica.cc:330] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.858165 23796 raft_consensus.cc:2229] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.858553 23796 raft_consensus.cc:2258] T 54bf83b19a6047a7bec69cda298d5436 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.860263 23796 tablet_replica.cc:330] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.860677 23796 raft_consensus.cc:2229] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.861402 23796 raft_consensus.cc:2258] T dc7a62fac3114417ae31d5ce8f6bbd86 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.863191 23796 tablet_replica.cc:330] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.863632 23796 raft_consensus.cc:2229] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.864300 23796 raft_consensus.cc:2258] T 8ffc860ae90a4860bc539207a9259897 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.865921 23796 tablet_replica.cc:330] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.866345 23796 raft_consensus.cc:2229] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:23:30.866734 23796 raft_consensus.cc:2258] T b4c99cde390d4c2981806647821fd09b P 948b43c3ee7f4e209c7c91300198700c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.868455 23796 tablet_replica.cc:330] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c: stopping tablet replica
I20240426 13:23:30.868849 23796 raft_consensus.cc:2229] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:30.869583 23796 raft_consensus.cc:2258] T 0013296c2f6d4bec82b9277e5df43e64 P 948b43c3ee7f4e209c7c91300198700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:30.933954 23796 tablet_server.cc:194] TabletServer@127.23.61.4:0 shutdown complete.
I20240426 13:23:30.988641 23796 master.cc:537] Master@127.23.61.62:34177 shutting down...
I20240426 13:23:31.012234 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:23:31.012781 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:23:31.013105 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P d342a68e43cc4fe09bbac496e74e2107: stopping tablet replica
I20240426 13:23:31.032979 23796 master.cc:552] Master@127.23.61.62:34177 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (31739 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
/home/jenkins-slave/workspace/build_and_test@3/src/kudu/master/auto_leader_rebalancer-test.cc:463: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[  SKIPPED ] LeaderRebalancerTest.TestMaintenanceMode (6 ms)
[----------] 5 tests from LeaderRebalancerTest (353296 ms total)

[----------] Global test environment tear-down
[==========] 5 tests from 1 test suite ran. (353297 ms total)
[  PASSED  ] 3 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] LeaderRebalancerTest.TestMaintenanceMode
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20240426 13:23:31.131259 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/tserver/ts_tablet_manager.cc:574: suppressed but not reported on 44 messages since previous log ~19 seconds ago
I20240426 13:23:31.131450 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/tserver/ts_tablet_manager.cc:522: suppressed but not reported on 44 messages since previous log ~20 seconds ago
I20240426 13:23:31.131596 23796 logging.cc:424] LogThrottler Service Queue Overflow: suppressed but not reported on 9 messages since previous log ~180 seconds ago
I20240426 13:23:31.131736 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/rpc/service_pool.cc:135: suppressed but not reported on 1 messages since previous log ~177 seconds ago
I20240426 13:23:31.131871 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/consensus/raft_consensus.cc:659: suppressed but not reported on 1 messages since previous log ~194 seconds ago
I20240426 13:23:31.132025 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 3 messages since previous log ~85 seconds ago
I20240426 13:23:31.132656 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 80 messages since previous log ~1 seconds ago
I20240426 13:23:31.132820 23796 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test@3/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 174 messages since previous log ~30 seconds ago